Source-linked AI summary
RuleWeaver: Benchmarking Rule-Centered Scenario Reasoning for Large Language Models
Bohan Yu, Shi-Yang Li, Pengfei Cao, Jun Zhao, Kang Liu
TL;DR
Existing benchmarks only partially evaluate whether LLMs can reason over complex rules in concrete scenarios, especially when rules have distinct roles. RuleWeaver constructs progressively complex rule-centered QA instances from corpus-derived Meta Rules and evaluates both answers and reasoning processes. Across 11 LLMs, performance remains limited, particularly under cross-source composition and exception or priority reasoning.
Problem
Existing benchmarks focus on output-level instruction constraints or provide rules in advance without sufficiently evaluating rule roles in concrete scenario reasoning.
Method
RuleWeaver progressively augments corpus-derived IF-THEN Meta Rules into complex rules, composes them into scenario QA, and scores rubric quality, rule recall, and rule precision.
Results
53.83 is the best rubric score in same-source evaluation and 50.27 in cross-source evaluation, with exception reasoning the weakest semantic category at 34.6%.
Takeaways & Limitations
Rule-centered scenario reasoning remains a key challenge for current LLMs, especially with cross-source composition, exception and priority reasoning, longer chains, and larger rule pools.
Takeaways & Limitations
RuleWeaver contains 96 English scenario QA instances from four corpora and uses an explicit rule pool with static single-turn QA.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly applied to specialized domains, where effective use of domain expertise often requires reasoning over complex rules in concrete scenarios. However, existing benchmarks only partially evaluate this capability, as they either focus on output-level instruction constraints or overlook the distinct roles that rules play in scenario reasoning. To address these gaps, this paper introduces RuleWeaver, a benchmark construction framework for evaluating rule-centered scenario reasoning. RuleWeaver starts from corpus-derived IF-THEN Meta Rules, progressively augments them into complex rules, and composes these rules into rule-centered scenario QA instances. Beyond final-answer correctness, RuleWeaver further supports process-level evaluation through rubric-based answer quality, rule recall, and rule precision. Experiments on 11 representative LLMs show that current models still struggle with complex rule-centered scenario reasoning, with even the best-performing model achieving only around 50% of the maximum rubric score. We make our code and dataset available here: https://github.com/SharkSpicy-NLP/RuleWeaver.
1 Introduction
RuleWeaver targets rule-centered scenario reasoning, where models must identify, apply, and justify rules with different roles in concrete scenarios. It constructs complex-rule QA instances and evaluates both answers and reasoning processes, finding substantial weaknesses across 11 LLMs.
- Rule-centered scenario reasoning requires identifying relevant rules, applying them under constraints, and justifying decisions through explicit rule-based reasoning.
- Existing instruction-following benchmarks primarily evaluate output constraints, while logic benchmarks usually provide rules in advance and overlook exceptions, conflicts, and priorities.
- 53.83 and 50.27 are the best rubric scores in same-source and cross-source settings, respectively, while the weakest rubric dimension reaches only 14.2% normalized score.
- RuleWeaver builds fine-grained complex rules from corpus-derived Meta Rules and composes them into rule-centered scenario QA instances.
- RuleWeaver evaluates process-level performance with rubric-based answer quality, rule recall, and rule precision, beyond final-answer correctness.
2 Related Work
Existing instruction-following and logical reasoning benchmarks cover complementary aspects of rule-related evaluation but leave scenario-centered, process-level reasoning underexplored. RuleWeaver addresses this gap by combining complex rule roles, multi-source composition, and fine-grained process evaluation.
- Instruction-Following Benchmarks: Instruction-following benchmarks test compliance with natural-language constraints such as format and style requirements.
- Logical Reasoning Benchmarks: Logical reasoning benchmarks use rules as premises for inference and may annotate logic or control reasoning depth.
- Logical Reasoning Benchmarks: Most existing benchmarks underexplore process-level scoring and multi-source or cross-domain rule composition.
- Logical Reasoning Benchmarks: Existing benchmarks often treat rules as homogeneous premises, limiting evaluation of rule selection, role distinction, and dependency-aware application.
- RuleWeaver bridges these benchmark lines by constructing role-diverse complex rules and concrete scenarios that support fine-grained process evaluation.
3 RuleWeaver
RuleWeaver is a traceable pipeline that transforms corpus-derived atomic Meta Rules into progressively augmented complex rule groups and dependency-aware scenario QA. Its generated instances expose rule retrieval and application processes for rubric, recall, and precision scoring.
- Framework and Rule Construction: A Meta Rule is an atomic IF-THEN rule with one triggering condition and one outcome, serving as the unit for progressive augmentation.
- Framework and Rule Construction: Rule augmentation varies semantic enhancement type, modification position, and logical combination method across six enhancement categories.
- Framework and Rule Construction: 11,145 extracted Meta Rules from four corpora are filtered into 200 high-quality rules preserving atomic IF-THEN structure.
- Framework and Rule Construction: Five-round progressive augmentation yields 200 traceable complex rule groups covering all six enhancement types.
- Scenario QA Construction: Scenario generation uses dependency planning, sub-scenario generation, final synthesis, and iterative quality review.
- Scenario QA Construction: Each instance samples five relevant rules from distinct groups under same-source or cross-source composition settings.
- Scoring Design: Evaluation reports rubric-based answer quality, rule recall, and rule precision to distinguish answer quality, retrieval, and rule application.
- Quality Assurance: Human quality assurance checks rule atomicity, clarity, traceability, logical consistency, scenario coherence, annotations, and rubrics.
4 Evaluation
RuleWeaver evaluates 11 LLMs on complex rule-centered scenarios using rubric quality, rule recall, and rule precision, revealing weaknesses across source composition, semantic roles, question types, complexity, and error patterns.
- Evaluation Setup: 11 models were evaluated on 96 complex scenario QA instances with five gold rules per instance and visible pools containing 200 complex rules.Evaluation disables configurable reasoning modes and generally uses temperature 0, with documented exceptions for Kimi-K2.6 and Gemini-3.1-Pro-Preview.
- Overall Performance: 53.83 was GPT-5.5’s highest same-source rubric score, while 50.27 was Claude-Opus-4.6’s highest cross-source score.Same-source leaders also include GPT-5.5 for precision (74.58) and Kimi-K2.6 for recall (72.92); cross-source leaders include Claude-Opus-4.6 for recall (62.50) and GPT-5.4 for precision (78.64).
- Overall Performance: Cross-source composition reduces average recall by 11.93 points and rubric score by 4.05 points, while increasing precision by 2.42 points.The reported averages cover all 11 evaluated models.
- Semantic Capabilities: Exception is the weakest semantic category at 34.6%, while conflict is strongest at 42.7%; Claude-Opus-4.6 leads five of six semantic axes.The paper links the low exception score to brittleness when applicability depends on exception structure rather than direct condition matching.
- Question Types: Average rubric scores are highest for Special-Case Judgment (43.5) and Definitive Conclusion (43.4), but lowest for Priority Arbitration (28.4) and Action Prescription (31.8).Different models lead different question types, indicating capability varies with the reasoning demanded by the scenario.
- Rubric and Error Analysis: Dependency Chain Alignment is the weakest rubric dimension at 14.2%, while multi-step integration is the most frequent error type at 86.7%.Other weak dimensions include Intermediate Conclusion Quality (21.0%), Issue Decomposition (28.4%), and Exception/Conflict Handling (30.4%); rule selection errors occur in 72.2% of answers.
5 Conclusion
RuleWeaver introduces a benchmark framework that converts corpus-derived Meta Rules into complex rule-centered scenario QA instances and evaluates both answer quality and rule use. Experiments across 11 LLMs show persistent limitations in complex rule-centered scenario reasoning.
- RuleWeaver transforms corpus-derived Meta Rules into complex rule-centered scenario QA instances.
- The benchmark evaluates rubric-based answer quality, rule recall, and rule precision beyond final-answer correctness.
- Experiments on 11 LLMs reveal limitations in complex rule-centered scenario reasoning, especially for cross-source composition, exception and priority reasoning, longer chains, and larger rule pools.
Limitations
RuleWeaver is a controlled benchmark whose current scope is limited in data coverage, interaction format, and rule-retrieval setting. The paper identifies broader multilingual, domain-specific, institutional, interactive, and changing-rule environments as future expansion areas.
- RuleWeaver contains 96 scenario QA instances built from four English corpora, limiting current coverage of multilingual, domain-specific, and institutional rule settings.
- The evaluation uses an explicit rule pool and static single-turn QA format rather than long-document retrieval, user interaction, or updates to changing rule sets.
- Future extensions target larger rule collections, unstructured-document retrieval, interactive settings, multilingual scenarios, and alternative judging protocols.
A Data Statistics
RuleWeaver organizes 200 complex rule groups from four balanced corpora and generates 96 scenario QA instances, each grounded in five complex rules. The dataset spans six augmentation types, balanced source settings, varied lengths, and seven question types.
- Complex rule set: The final rule set contains 200 complex rule groups, with 50 groups each from GovReport, WikiHow, CUAD, and BookSum.
- Complex rule set: Each rule group has four final variants: one moderate continuation and variants for EXCEPTION, CONFLICT, and IRONCLAD enhancements.
- Scenario QA set: The scenario QA set contains 96 instances, each grounded in exactly five complex rules, with 48 cross-source and 48 same-source instances.
- Scenario QA set: The scenarios contain 72,291 words overall, averaging 753.0 words per instance, with lengths ranging from 154 to 1,499 words.
- Semantic augmentation: Rule augmentation includes ABSTRACT, ADDITIVE, NEGATE, EXCEPTION, CONFLICT, and IRONCLAD, which modify granularity, scope, polarity, applicability, compatibility, or priority.
B.2 Implementation Details for Complex Rule Construction
Complex rule construction combines corpus sampling, Meta Rule generation and filtering, progressive semantic augmentation, dependency-based scenario generation, and multi-stage quality evaluation. The implementation also uses structured audits and bootstrap analyses to assess judging and result reliability.
- Rule construction: Documents are embedded with Qwen3-8B-Embedding, clustered with KMeans into 50 clusters per dataset, and sampled through nearest-document selection.
- Rule construction: Chunks of up to 3,000 tokens generate 10 Meta Rules each with GPT-5.4, followed by atomicity verification and filtering of weak or externally dependent rules.
- Rule construction: Meta Rules use an atomic IF-THEN format with one triggering condition and one outcome, and serve as seeds for progressive complex-rule augmentation.
- Scenario generation: Dependency planning selects among seven question types and builds a directed rule-application structure linking intermediate conclusions to later reasoning steps.
- Evaluation: The answer judge uses a 100-point, nine-dimension rubric separating surface validity from decomposition, dependency alignment, and exception or conflict handling.
- Reliability analysis: Bootstrap confidence intervals resample 48 aligned QA instances 20,000 times, with bootstrap means differing from point estimates by at most 0.43 points.
C.2 Semantic Enhancement Results under the 200-Rule Setting
Under the 200-rule setting, Claude-Opus-4.6 is strongest overall across semantic enhancement types, while EXCEPTION remains especially difficult for models.
- The evaluation measures semantic rule following through event-level correct-application rates across six enhancement types.
- Claude-Opus-4.6 achieves the best scores on ABSTRACT, ADDITIVE, NEGATE, and CONFLICT, and ties GPT-5.5 on IRONCLAD.
- The rubric dimensions total 100 points, with bootstrap analyses reporting mean scores and 95% confidence intervals.
- Claude-Sonnet-4.6 performs best on EXCEPTION, indicating comparatively strong handling of special-case carve-outs.
- CONFLICT and IRONCLAD are the highest-scoring semantic types across models, while EXCEPTION, NEGATE, and ADDITIVE are more challenging.
C.3 Question-Type Results under the 200-Rule Setting
Question-type results under the 200-rule setting show uneven difficulty across scenario tasks, with rubric performance localized by question type and model.
- Figure 5 breaks down normalized rubric scores by model to localize where models lose points during rule-centered reasoning.
- Dependency-chain alignment remains low even as stronger models improve on rule-grounded reasoning and final-answer consistency.
C.5 QA-Level Spearman Correlation Analysis
QA-level correlations show that correctly applying cited rules is more closely associated with answer quality than merely recalling more required rules.
- The analysis computes Spearman rank correlations over QA-level structural, rule-selection, and rubric-score values.
- Rule precision correlates strongly with mean rubric score at ρ = 0.66, whereas rule recall has a weaker correlation of ρ = 0.20.
- Reasoning depth correlates negatively with rubric score at ρ = −0.29, and dependency load correlates at ρ = −0.22.
- Structural branching is nearly uncorrelated with rubric score at ρ = −0.01.
C.6 Case Studies
The case studies document recurring rule-centered reasoning failures, including missing rules, reversed exceptions, and broken dependency chains, alongside prompts and rubric-based scoring procedures.
- Case Studies: Representative failures include missing necessary rules, misusing exception conditions, and failing to propagate intermediate conclusions across dependent rules.
- Recall Missing: A missed aggregation rule can leave determinate obligations unresolved, causing the model to treat them as pending or contingent.
- Rule Misuse: Reversing an exception condition can produce the opposite consequence, such as incorrectly concluding that no interest accrues.
- Dependency Chain: Failure to propagate an intermediate conclusion across dependent rules can make locally plausible applications fail to form the required global reasoning chain.
- Scoring Cases: Figures 9–17 provide zero-score examples for all nine rubric dimensions, pairing model responses with evidence, reference answers, and judge rationales.