PaperlayerView exact revision on arXiv

Source-linked AI summary

RuleTrek: Benchmarking Agentic Rule Reasoning in Complex Scenarios

Chenhao Yuan, Bohan Yu, Yinhao Xu, Chao Yin, Chenxi Zhou, Zhaoxuan Wang, Shuwen Xu, Shaoping Huang, Haolin Ren, Jiaxiang Liu, Yao Xu, Jun Zhao, Kang Liu

arXiv:2610.19698v1cs.AI

TL;DR

Existing benchmarks often simplify scenarios or rarely test applying domain-specific rules to complex, distributed information. RuleTrek constructs rule-grounded scenarios and evaluates three kinds of rule reasoning; experiments on 11 frontier LLMs show substantial challenges across them.

  • Problem

    Existing rule-reasoning benchmarks often simplify scenarios, and realistic-workspace benchmarks rarely assess identifying and applying domain rules in complex, multi-file scenarios.

    Source: §1 · p. 1

  • Method

    RuleTrek constructs scenarios from structured rule graphs by generating rule-specific sub-scenarios and composing them with consistency and solvability checks, then evaluates RI, RD, and E2E in interactive workspaces.

    Source: §1 · p. 2

  • Results

    Across 11 frontier LLMs, RI scores peak at 62.47%, RD at 60.43%, and E2E scores range from 26.97% to 55.29%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Experiments reveal substantial difficulties identifying and applying rules to complex scenarios, even when verified rules are provided.

    Source: §5 · p. 9

    RuleTrek is limited to English-language rule repositories and scenarios and uses predefined repositories for controlled, reproducible evaluation.

    Source: §6 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) increasingly tackle complex real-world tasks in specialized domains. However, identifying and correctly applying domain-specific rules remains challenging when scenario information is distributed across multiple files. Existing rule-reasoning benchmarks typically represent scenarios using only a few factual statements, simplifying the identification and application of relevant rules. To address this gap, this paper introduces \textbf{RuleTrek}, a multi-domain benchmark for agentic rule reasoning in complex scenarios. Its rule-grounded construction pipeline extracts structured rule graphs, generates and incrementally composes rule-specific sub-scenarios while ensuring consistency and solvability. To systematically evaluate agentic rule reasoning in complex scenarios, RuleTrek assesses three capabilities in interactive workspaces: Rule Identification (RI), Rule Deduction (RD), and End-to-End Rule Reasoning (E2E). Experiments on 11 frontier LLMs show that the highest RI and RD scores are only 62.47\% and 60.43\%, respectively, while E2E scores range from 26.97\% to 55.29\%. These results demonstrate that current LLM agents face substantial challenges in rule reasoning over complex scenarios. We make our code and dataset publicly available at \href{https://github.com/yuan247/RuleTrek}{RuleTrek}.

1 Introduction

RuleTrek addresses limited evaluation of rule reasoning in complex, multi-file scenarios with a benchmark and a rule-grounded construction pipeline. Across 11 frontier LLMs, results show substantial difficulty identifying and applying rules in these scenarios.

1 Introduction

  • Existing rule-reasoning benchmarks often use a few facts, while realistic-workspace benchmarks rarely test identifying and applying domain rules to complex scenarios.Complex tasks may require combining taxpayer profiles, salary records, and royalty income across files to apply rules about residency, tax rates, and deductions.Source: §1 · pp. 1–2
  • RuleTrek is a multi-domain benchmark that evaluates agentic rule reasoning in complex scenarios using interactive workspaces.Agents iteratively explore rule documents and scenario files, retrieve relevant information, and use tools as needed.Source: §1 · p. 2
  • RI scores peak at 62.47%, RD at 60.43%, and E2E ranges from 26.97% to 55.29% across 11 frontier LLMs.Rule Identification (RI) tests finding rules and dependencies; Rule Deduction (RD) tests applying provided rules; End-to-End Rule Reasoning (E2E) tests both.Source: §1 · p. 2
  • Its construction pipeline extracts rule graphs, generates rule-specific sub-scenarios, and composes them while checking consistency and rule-based solvability.Sub-scenarios are composed according to rule dependencies, with consistency and solvability verified at each composition step.Source: §1 · p. 2

2 Related Work

Prior rule-reasoning benchmarks and realistic-workspace agent benchmarks cover complementary capabilities, but each leaves a gap in evaluating rule reasoning over complex, distributed scenarios. RuleTrek addresses that gap, and Table 1 compares benchmark features and shows its domain distribution.

2 Related Work

  • RuleTaker studies deduction over normalized rules and facts (Clark et al., 2020), while PolicyQA and LegalBench extend rule reasoning to real-world policies and legal domains (Guha et al., 2023).More recent examples include RuleArena, RuleWeaver (Yu et al., 2026b), and SearchFireSafety (Chae et al., 2026).Source: §2 · p. 2
  • Table 1 compares benchmarks on rule application, native rules, distributed scenarios, multiple domains, and interactive workspaces, alongside RuleTrek's instance distribution across six domains.Its caption defines the comparison criteria and the table symbols ✓, ✗, and ⊖ as full, absent, and partial support.Source: Table 1 · p. 3
  • Rule-reasoning benchmarks often simplify scenarios, while workspace benchmarks rarely assess identifying and applying domain rules in complex scenarios.RuleTrek evaluates rule identification, deduction, and end-to-end reasoning in complex multi-file scenarios.Source: §2 · p. 3

3 RuleTrek Construction

RuleTrek builds complex scenarios from verified rule graphs by generating rule-specific sub-scenarios and composing them according to rule dependencies. Human checks at successive stages support scenario consistency, solvability, and answerability.

3 RuleTrek Construction

  • The construction pipeline extracts and verifies rule graphs, generates rule-specific sub-scenarios, then composes them with consistency and solvability checks.LLM-based construction steps use GPT-6-Astra (OpenAI, 2026c).Source: §3 · p. 3, Figure 2 · p. 4

3.1 Stage 1: Domain Rule Graph Extraction

  • For each domain, RuleTrek extracts a structured rule graph from real-world repositories and manually verifies rules, source references, and dependencies.Repositories span six domains and retain their original formats; graph edges capture relationships such as dependencies, exceptions, overrides, precedence, and temporal constraints.Source: §3.1 · p. 3

3.2 Stage 2: Rule-Specific Sub-Scenario Construction

  • For each graph rule, an LLM generates a sub-scenario that contains the information needed to apply that rule and requires its application to resolve.Sub-scenarios use domain-appropriate files and are manually reviewed for consistency, solvability, and sufficient information.Source: §3.2 · p. 4

3.3 Stage 3: Dependency-Aware Composition

  • RuleTrek incrementally composes sub-scenarios according to rule dependencies, revising files as needed to keep information coherent across them.Human experts verify intermediate scenarios for consistency and solvability; composition proceeds only after both checks pass, while the underlying rule graph remains unchanged.Source: §3.3 · pp. 4–5

3.4 Question Instantiation

  • After the final scenario passes checks, experts solve each generated question using the original rules and files, verifying answerability, uniqueness, and rule compliance.Invalid instances are revised and revalidated.Source: §3.4 · p. 5

3.5 Three Evaluation Capabilities

  • RI tests identifying rules and dependencies, RD tests applying a provided verified graph, and E2E requires both rule identification and application.All three capabilities use the same complex scenarios and questions in interactive workspaces, with different rule information provided and required outputs.Source: §3.5 · p. 5

4 Experiments

Across six domains, the experiments evaluate 11 frontier LLMs on rule identification, rule deduction, and end-to-end reasoning, finding substantial challenges in all three. They also examine error patterns, rule-graph complexity, agent behavior, and scoring reliability.

4.1 Experimental Setup

  • Eleven frontier LLMs were evaluated across 240 instances in six domains, using isolated workspaces with file and shell access and no external network.Each model ran three times under a 30-minute and 200-step limit; scores combined answer correctness with reasoning quality and were averaged over instances.Source: §4.1 · pp. 5–6, Table 3 · p. 6

4.2 Main Results

  • GPT-6.1-Sol achieved the highest average RI score (62.47%) and RD score (60.43%), while E2E scores ranged from 26.97% to 55.29%.The results indicate difficulty both identifying applicable rules and integrating distributed scenario information to apply them.Source: Table 2 · p. 6, §4.2 · p. 6

4.3 Error Analysis of Rule Identification

  • RI errors included missed condition-specific controls and supporting rules, plus incorrectly merged rules with distinct applicability conditions.The analysis identifies mapping scenario evidence to rules, finding rule dependencies, and distinguishing applicability conditions as key challenges.Source: §4.3 · pp. 6–7, Table 4 · p. 6

4.4 Performance Degrades with Rule-Graph Complexity

  • Mean E2E scores fell from 76.5 to 23.6 as rule-node count rose from at most 3 to at least 14, and from 62.3 to 26.4 as depth rose from at most 1 to at least 5.The reported trends associate larger rule graphs and longer dependency chains with greater end-to-end reasoning difficulty.Source: §4.4 · p. 7, Figure 3 · p. 7

4.5 Complex Scenarios Challenge Both Rule Identification and Application

  • RD outperformed E2E by 3.79–8.15 points, yet performance remained limited even when agents received the verified rule graph.For GPT-6.1-Sol, rule-selection errors were the largest failure category, accounting for 54.9% of failed trajectories.Source: §4.5 · p. 7, Figure 4 · p. 7

4.6 Model Differences in Agent Behavior and E2E Performance

  • GPT-6.1-Sol achieved the highest E2E score (55.29%) with relatively few steps, while more steps did not consistently correspond to higher scores.In a game example, GPT-6.1-Sol tracked a consumed ward correctly in 34 steps, whereas Qwen3.8-Flash treated it as active and answered incorrectly after 57 steps.Source: §4.6 · p. 8, Figure 5 · p. 8

4.7 Scoring Reliability

  • Human–judge Spearman correlations ranged from 0.822 to 0.857, below human–human agreement of 0.933–0.951.Across alternative judge families, submission-level correlations were 0.903–0.934 and model-ranking correlations were 0.955–0.991.Source: §4.7 · p. 8, Table 5 · p. 8

5 Conclusion

RuleTrek evaluates agentic rule reasoning in complex scenarios through three capabilities and a rule-grounded construction pipeline. Results show substantial difficulty identifying and applying rules, even when verified rules are provided.

5 Conclusion

  • RuleTrek extracts rule graphs, composes rule-specific scenarios, and evaluates Rule Identification, Rule Deduction, and End-to-End Rule Reasoning in interactive workspaces.The pipeline is designed to ensure scenario consistency and solvability.Source: §5 · p. 8
  • LLMs face substantial difficulty identifying and applying rules to complex scenarios, even when verified rules are provided.Source: §5 · p. 9

6 Limitations

RuleTrek uses English-language repositories and scenarios, and predefined rule and scenario repositories, for controlled and reproducible evaluation.

6 Limitations

  • RuleTrek's repositories and scenarios are English-language, a choice made to maintain a consistent evaluation setting.Source: §6 · p. 9
  • The benchmark uses predefined rule and scenario repositories to enable controlled and reproducible evaluation.Source: §6 · p. 9
  • Future extensions could explore multilingual settings and dynamic environments where rules and scenario information evolve over time.Source: §6 · p. 9

A Benchmark Quality Verification

RuleTrek's benchmark instances undergo human verification for answerability, answer uniqueness, and fidelity to the rules and scenarios. Two domain-knowledge annotators independently review each instance across five quality criteria.

A Benchmark Quality Verification

A.1 Human Verification Protocol

  • Two domain-knowledge annotators independently solve each question and assess answerability, uniqueness, rule alignment, scenario consistency, and difficulty validity.Instances are checked against original rule repositories, verified graphs, scenario files, and instantiated questions.Source: §A.1 · pp. 10–11
  • Table 6 reports Cohen’s κ and initial pass rates for each quality criterion before revisions.Source: Table 6 · p. 11

A.2 Verification Results

Verification results report initial agreement and pass rates before revision, followed by iterative checking until all instances satisfy the quality requirements.

A.2 Verification Results

  • Table 6 reports initial verification results, with Cohen’s κ measuring annotator agreement and pass rates reflecting the first independent review.Failed instances were revised and rechecked in later quality-control rounds.Source: §A.2 · p. 11, Table 7 · p. 11

A.3 Multi-round Quality Control

  • Instances failing any verification criterion are revised and independently checked again until all instances satisfy the requirements.Table 7 reports the pass rate after each verification round.Source: §A.3 · p. 11, Table 7 · p. 11

B Additional Experimental Analysis

This section examines whether generation length tracks end-to-end rule reasoning and analyzes what successful performance requires. Across evaluated models, longer outputs show no consistent association with higher E2E scores.

B.1 Output Length and E2E Performance

  • There is no consistent correlation between average output tokens per task and E2E scores across evaluated models.Figure 6 plots this relationship, with each point representing one model.Source: §B.1 · p. 11, Figure 6 · p. 12
  • Several models produce substantially more tokens yet achieve lower E2E scores than models with shorter outputs.The results indicate that output length alone is not a reliable marker of end-to-end performance.Source: §B.1 · p. 11
  • Successful E2E reasoning depends on retrieving relevant information, identifying applicable rules, and correctly applying rule dependencies rather than primarily producing longer responses.Source: §B.1 · p. 11

B.2 Detailed Error Analysis of Rule Identification

Rule identification failures arise from scenario structures that require reasoning over distributed evidence and rule dependencies, not just semantic matching. Paired trajectories also illustrate failures in maintaining state and retrieving prerequisite rules.

B.2 Detailed Error Analysis of Rule Identification

  • Rule identification failures across six domains reflect heterogeneous evidence, hidden dependencies, and stateful interactions spread across multiple artifacts.Success requires linking scenario evidence to rule conditions, recovering prerequisites, and distinguishing similarly described rules with different applicability.Source: §B.2 · p. 12
  • These cases show that rule identification requires reasoning over scenario structure and dependencies rather than relying only on semantic similarity.Source: §B.2 · p. 12

C Paired Trajectory Case Studies

  • The paired case studies compare a rescue task’s state updates and a tax scenario’s missing prerequisite rule, using observed tool interactions and submitted results rather than hidden reasoning.Source: C Paired Trajectory Case Studies · p. 12

C.1 Cross-Model E2E Comparison

A rescue-task comparison illustrates that retrieving relevant rules is not enough: agents must propagate their effects through later events and maintain consistent scenario state.

C.1 Cross-Model E2E Comparison

  • GPT-6.1-Sol correctly consumes the ward after the first trigger and completes the six-patient task in 34 steps with an E2E score of 100, while Qwen3.8-Flash retains the ward and scores 0 after 57 steps.The inconsistent state update affects later death-save and revival events; additional validation does not correct the trajectory.Source: §C.1 · p. 12
  • Both agents retrieve the protection rule, but only GPT-6.1-Sol maintains the updated protection state across subsequent events.Source: §C.1 · p. 12
  • The example demonstrates that successful rule retrieval alone is insufficient; agents must propagate rule effects and keep state consistent across distributed records.Source: §C.1 · p. 12

C.2 Same-Model RD–E2E Comparison

The paired tax comparison shows how access to an explicit rule graph changes prerequisite-rule retrieval, while scenario-grounded application remains necessary. The accompanying prompts specify construction, task, review, failure-annotation, and scoring procedures.

C.2 Same-Model RD–E2E Comparison

  • With the rule graph, Qwen3.8-Flash applies prerequisite netting and gets $15,644 in six steps (RD score 97.75); without it, E2E misses the rule, gets $16,564 in 15 steps, and scores 0.The graph reduces retrieval burden but does not eliminate scenario-grounded application.Source: §C.2 · pp. 12, 14, Figure 8 · p. 15

D.1 Benchmark Construction Prompts

  • The prompts extract source-grounded rule graphs, generate rule-specific artifacts with evaluable conditions, compose scenarios while preserving dependencies and consistency, and revise issues identified by experts.Question instantiation requires objectively checkable answers determined by the scenario and rules, without revealing operative rules or the answer.Source: §D.1 · pp. 16–17, Question instantiation · p. 17

Question instantiation.

  • Question-generation instructions require a concrete, unambiguous question and a private solvability note identifying the necessary evidence.Source: Question instantiation · p. 17

D.2 Agent Task Instructions

  • In isolated workspaces, RI identifies operative rules and relations, RD applies a supplied graph to distributed scenario files, and E2E discovers rules from original sources before solving.Runs have file and shell access, no external network access, a 30-minute and 200-step limit, and JSON submissions.Source: §D.2 · pp. 17–18

D.3 Human Review and Scoring Instructions

  • Human review checks rule-graph fidelity, scenario solvability and consistency, composition, and diversity; submission scoring reliability is also studied with human annotators using the judge rubrics.Source: §D.3 · p. 19

D.4 Failure-type Annotation Instructions

  • E2E failures are assigned one primary category at the earliest decisive error: retrieval miss, rule selection, reasoning error, or final synthesis.Annotations use observable interactions, accessed files, answers, and reference evidence rather than inferred hidden reasoning.Source: §D.4 · p. 19

D.5 Automatic Evaluation Judge Prompts

  • RI scores average two rubric dimensions, while RD and E2E reasoning scores average four; exact-answer correctness is checked separately by deterministic code.The derivation judge assesses four dimensions on a 0–100 scale.Source: §D.5 · p. 19, Figure 10 · p. 21

Related papers

Loading 2610.19698v1…