Source-linked AI summary
Grounded Normative Rule Generation with Structured Search
Fanqi Kong, Huaxiao Yin, Ruijie Zhang, Xiaoyuan Zhang, Yizhe Huang, Jian Gao, Shuo Chen, Song-Chun Zhu
TL;DR
Normative rule generation must produce readable rules that are operationally verifiable against available records, a requirement not enforced by many existing generation and structured-output methods. The paper formalizes GNRS and uses GNRS-Search to search five-slot operational structures before rendering prose, raising average rubric quality from 68.8% to 81.0% across its evaluations. The results support inspectable, operationally grounded rule drafting, while the benchmarks remain limited in policy breadth and long-document interpretation.
Problem
GNRS addresses the gap between readable normative rules and rules strictly backed by available institutional records.
Method
GNRS-Search uses MCMC to search a discrete five-slot And-Or Graph for scope, trigger, norm, evidence, and procedure before rendering natural-language text.
Results
68.8% to 81.0%: GNRS-Search improves average rubric quality on the main test and ranks first under the overall executability metric across model backbones and out-of-distribution scenes.
Takeaways & Limitations
The framework makes rule drafting an inspectable search problem and supports verifiable, auditable policy generation within the evaluated scope.
Takeaways & Limitations
GNRS-Bench is controlled, RealCharter-Bench uses normalized policy cards rather than complete source documents, and incorrect schemas remain a sensitivity boundary.
Abstract
from arXiv · showhide
Normative rules like institutional charters and workplace policies must be both human-readable and operationally verifiable against actual environment records. However, current language generation and structured-output benchmarks primarily reward surface fluency or schema compliance, leaving operational grounding weakly tested. This creates a critical vulnerability where standard language models generate plausible-sounding policies that fail during enforcement because they rely on unavailable data logs or misaligned scopes. To address this challenge, we formalize the problem as Grounded Normative Rule Synthesis (GNRS) and introduce GNRS-Search, a framework that utilizes Markov Chain Monte Carlo (MCMC) sampling to optimize a discrete, five-slot And-Or Graph (AOG). By explicitly decoupling intermediate operational structure from final prose generation, this method isolates executable feasibility from writing style and allows rule failures to be localized prior to surface realization. We evaluate our approach on GNRS-Bench, a benchmark spanning 116 controlled goals across eight scene families, and RealCharter-Bench, which evaluates transfer to 53 real-derived policy tasks with hidden source clauses. GNRS-Search raises average rubric quality from 68.8% to 81.0% and ranks first under a disclosed executable composite metric, while systematic slot interventions confirm that performance gains stem from robust operational logic rather than rhetorical tuning. Ultimately, by transforming automated rule drafting into an inspectable search problem, this work provides a foundational paradigm for deploying verifiable and compliance-ready personal agents within regulated environments.
1 Introduction
GNRS frames normative rule writing as producing readable rules that are also verifiable against available records. GNRS-Search addresses this by searching operational structures before rendering prose, improving benchmarked rule quality and operational validity.
- Motivation: Normative rules must be human-readable and operationally verifiable, but fluent clauses can rely on unavailable evidence or untraceable procedures.The task therefore requires coordinating rule scope, trigger, norm, evidence, and procedure.
- Motivation: Existing constrained-generation and self-refinement methods improve syntax or feedback-driven output quality without enforcing grounding in available records.Interactive agents use environmental feedback for action selection, whereas GNRS evaluates whether the rule text itself is backed by auditable logs.
- Approach: GNRS-Search represents rules as five-slot And-Or Graphs covering scope, trigger, norm, evidence, and procedure before generating final text.The candidate space combines templates, schemas, record logs, and language-model proposals, while MCMC edits the discrete structure.
- Evaluation: 116 controlled goals across 8 scene families and 53 real-derived policy tasks form the GNRS-Bench and RealCharter-Bench evaluation settings.The benchmarks test main, robustness, out-of-distribution, and real-derived performance.
- Results: 68.8% to 81.0%: GNRS-Search raises average rubric quality on the main test and ranks first under the overall executability metric.Human evaluations and sensitivity analyses attribute the gains to correct operational structure rather than better phrasing.
2 Related Work
Prior work addresses grounded realization, structured output, rule following, agent behavior, and symbolic search, but GNRS focuses on synthesizing enforceable normative language from goals and operational records.
- Grounded and constrained generation: Grounded NLG and constrained decoding improve faithful realization or parseability, but schema-level validity does not ensure operational grounding.Best-of-N selection and verifier-style reranking are also established related approaches.
- Rule grounding and agents: Rule-oriented benchmarks study rule following or evidence-grounded reasoning, whereas GNRS synthesizes the rule itself from goals and records.Interactive-agent research instead uses environmental feedback to guide behavior.
- Symbolic synthesis and constraints: GNRS-Search applies discrete structural search before natural-language generation, following the spirit of program synthesis without complete logical specifications.Its constraints are partial grounding diagnostics over available records rather than full formal specifications.
3 Preliminaries
The preliminaries model grounded rule generation as selecting a latent operational structure and then realizing it as natural language. This separates operational interpretation from surface fluency while retaining a structured validity objective.
- Problem formulation: A rule-writing context consists of normative intent, an institutional schema, and an operational record layer, producing a natural-language rule.Quality depends on preserving intent and allowing operationalization against the schema and records, not fluency alone.
- Problem formulation: The evaluation vector measures intent preservation, grounding, evidence use, procedural closure, and readability.A scalar task score is defined as a monotone functional of these components.
- Latent structure: The latent structure contains five slots: scope, trigger, norm, evidence, and procedure.It mediates between intent and surface text and provides a way to localize grounding failures.
- Structure search: The framework constructs a candidate space, searches for an operational structure a∗, and realizes it as text through a realization function.This explicitly separates operational interpretation from surface realization rather than estimating the final text in one step.
- Structure search: The internal structure score S remains distinct from the final-language evaluation score to avoid optimizing the same metric used for judgment.The formal proposition relates score calibration, approximate search, and realization loss to task validity.
4 Evaluation Benchmarks
The evaluation uses a controlled benchmark for verifiable rule synthesis and a real-derived benchmark for transfer. Instances constrain generation to institutional schemas and records rather than human-authored reference rules.
- GNRS-Bench: GNRS-Bench provides typed, systematically varied operational records for evaluating grounded rule synthesis in a controlled simulation environment.Each instance pairs a management goal with an institutional context schema and an operational record layer.
- GNRS-Bench: Valid GNRS-Bench rules must ground scope, trigger, evidence, and procedure in available logs instead of unrecorded channels or subjective testimony.An attendance example exposes check-in records through the record layer.
- GNRS-Bench: 116 goals are distributed across four functional categories, with 88 core in-distribution goals and a separate 12-goal open_space_navigation out-of-distribution split.The benchmark also reserves 16 goals for development tuning.
- Evaluation protocol: Testing withholds human-authored golden rules and scores each generated clause using a blinded multidimensional rubric against permitted context and logs.The protocol emphasizes interpretable semantic and programmatic commitments rather than reference-text imitation.
- RealCharter-Bench: RealCharter-Bench contains 53 tasks derived from 15 open-source governance guidelines and corporate manuals to test transfer to authentic policy materials.Long-document understanding and policy extraction are outside the benchmark’s current scope.
5 Method
GNRS-Search separates operational structure from prose, assembling grounded slot candidates and optimizing a five-slot AOG before surface realization. Its search combines MCMC exploration, grounding-aware scoring, feasibility checks, and guarded realization to localize structural errors before final text.
- Architecture: The pipeline constructs grounded operational candidates, searches complete five-slot AOG configurations, and renders the selected structure as normative prose.This separates structural commitments from surface realization and targets executable, auditable interpretations.
- Candidate construction: Candidate repositories combine symbolic templates, context-schema primitives, trace-derived records, and language-model proposals into a Cartesian-product structural search space.The framework validates language-model fragments against schemas and active logs before retaining them.
- Structured search: MCMC explores the discrete AOG space through slot replacement, parameter tuning, evidence swaps, and coordinated trigger-evidence edits.These proposal operations support exploration of fully populated five-slot structural layouts.
- Structured search: The internal scorer combines structural grounding and linguistic features while penalizing hard operational feasibility violations.Its features cover intent alignment, schema grounding, record compatibility, procedural closure, and value consistency; feasibility is checked by HARD-OK.
- Evaluator separation: Search weights are calibrated on 16 development goals, frozen before testing, and separated from the downstream validation metric.Appendix analyses alternative weightings, hard-constraint ablations, and a CP-SAT baseline to isolate structural-search bias.
- Realization and evaluator separation: A payload-preserving polish workflow and programmatic alignment guard prevent realization edits from omitting essential structural entities or required evidence logs.The optimized layout is first converted into an initial clause before guarded text refinement.
6 Results and Analysis
GNRS-Search achieves strong quality and executable performance across benchmark, robustness, and transfer evaluations, while exposing a quality–feasibility trade-off and sensitivity to incorrect schemas. Ablations and slot interventions indicate that structured search and semantic rule components, rather than surface phrasing alone, account for the gains.
- Main Benchmark Results: 81.0% Rubric Avg (v1) is achieved by GNRS-Search on the main test, with executability reaching 73.3%.The main test uses 88 goals across three seeds and reports Rubric Avg (v1) as the equal-weight mean of five rubric dimensions.
- Main Benchmark Results: GNRS-Search ranks highest on the disclosed OVERALL composite, while No LLM candidates attains higher HARD-OK.Both methods are nondominated: full GNRS-Search has the highest reader-facing quality, whereas the ablation has stronger standalone feasibility.
- Main Benchmark Results: 0.77 Spearman correlation and a 0.69 positive human score margin support alignment between the automatic scorer and method-blinded human calibration.The human executability audit separately found 81.3% strict executability for No LLM candidates, 75.0% for full GNRS-Search, and 68.8% overall human preference for full GNRS-Search.
- Ablations: Random candidate selection, shorter search, and removing the hard feasibility penalty each degrade performance, while CP-SAT remains below the full framework.These ablations indicate that candidate ranking, systematic search, and feasibility-aware scoring contribute to the observed results.
- Mechanism and Sensitivity: Norm-body corruption causes the largest average quality loss, followed by scope and trigger corruption, whereas procedure and evidence-policy slots are comparatively resilient.The intervention replaces one slot’s candidate pool with cross-goal distractors; examples show that scope, trigger, and norm changes alter the rule’s semantic commitments.
- Robustness and Transfer: 79.8% average quality and 82.7% joint composite are maintained on held-out open-space navigation, and GNRS-Search ranks first across tested generator backbones.Direct prompting can retain surface fluency but consistently receives zero binary feasibility scores in the reported comparison.
- Robustness and Transfer: 2.1-point, 5.4-point, and 9.7-point Rubric Avg (v1) drops follow irrelevant noise, missing information, and incorrectly specified schemas, respectively.The method is relatively stable under irrelevant noise and partial information but sensitive to incorrect operational inputs, making schema validation a deployment requirement.
- Robustness and Transfer: 93.4% average text quality with perfect feasibility is achieved on RealCharter-Bench, where structured optimization leads the strongest iterative self-refinement baseline.The transfer result is attributed to explicit target-slot enumeration within normalized task cards, which reduces unsupported environment payloads relative to open-ended simulation settings.
7 Conclusion
The paper formalizes grounded normative rule synthesis and introduces GNRS-Search, which separates structural commitments from prose generation through discrete MCMC sampling over an AOG. Evaluations and sensitivity audits support gains in operational meaning across benchmark and transfer settings.
- GNRS is formalized alongside GNRS-Bench and RealCharter-Bench to evaluate grounded normative rule synthesis.
- The evaluation includes external transfer from the synthetic benchmark to 53 real-derived charter tasks for three anchor methods.
- GNRS-Search decouples structural commitments from surface prose generation through discrete MCMC sampling over an AOG layout.
- The framework outperforms direct prompting and iterative self-refinement across generator backbones and out-of-distribution settings.
- Systematic sensitivity audits indicate that performance gains arise from securing policy semantic structure rather than optimizing superficial phrasing.
Limitations
The evaluation is bounded by controlled benchmark design, normalized policy inputs, limited evaluator coverage, and an assumption that operational vocabulary and evidence interfaces are available. Incorrect operational inputs remain a sensitivity boundary requiring independent validation.
- GNRS-Bench uses a fixed context schema and does not cover the full diversity of institutional policies, legal domains, languages, or deployment environments.
- RealCharter-Bench uses normalized policy cards rather than complete source documents, so it does not test long-document interpretation or end-to-end policy extraction.
- Rubric scores, audits, generator comparisons, and human calibration do not substitute for broad evaluator-family studies or domain-expert legal review.
- GNRS-Search assumes that relevant operational vocabulary and evidence interfaces are available before generation.
- The method is relatively stable under irrelevant noise and partial missing information but sensitive to incorrectly specified operational inputs.
AI Assistant Use
The authors report that all research work was conducted by the authors, while AI assistants were used only for language polishing and editing support.
- All research ideas, experiments, benchmark construction, evaluations, analyses, claims, and conclusions were developed and conducted by the authors.
- AI assistants were used only for language polishing and editing support.
A Benchmark Details
The benchmark artifacts document GNRS-Bench’s taxonomy, operational vocabulary, and evaluation splits alongside RealCharter-Bench’s public-policy provenance. The controlled benchmark contains 116 goals, while the real-derived benchmark contains 53 tasks from public policy documents.
- The appendix documents GNRS-Bench’s goal taxonomy and RealCharter-Bench’s document lineage for reproducible dataset composition.
- Benchmark artifacts convert public policy materials into normalized task cards with hidden source clauses and environment-derived record layers defining actions, observables, and evidence channels.
- GNRS-Bench contains 116 goals organized by scene family and goal category, with four categories containing 29 goals each.
- The record layer normalizes 286 child-action definitions, 37 NPC action names, 16 composite actions, 3,285 resource instances, and 38 rooms into generator-visible typed vocabulary.
- RealCharter-Bench contains 53 tasks drawn from 15 publicly available policy documents and grouped into four source families.
- The open_space_navigation family is held out for out-of-distribution evaluation, leaving 104 goals split into 16 development and 88 in-distribution test goals.
- Illustrative benchmark goals pair category examples with required and forbidden actions from the canonical action set.
- The operational vocabulary is fixed across methods and seeds, and the search space combines these primitives with goal-level templates.
B Extended Results
The extended results expose trade-offs behind GNRS-Search’s headline performance, including its composite-score advantage despite a standalone HARD-OK disadvantage. RealCharter-Bench uses normalized policy cards while withholding original clause text from generators.
- Extended results: GNRS-Search consistently wins on the disclosed OVERALL composite while losing on the standalone binary HARD-OK gate.The contrast reflects a trade-off between composite performance and standalone feasibility.
- Evaluation coverage: RealCharter-Bench exposes 53 tasks as normalized policy cards while withholding the original clause text from generators.Each card includes source family, domain, context, roles, triggers, normative summaries, actions, evidence channels, procedures, and expected slots.
- Extended results: Table 10 summarizes the key contrasts across the result suite, with positive gaps favoring the first method or condition and negative gaps preserving trade-offs.The appendix expands these contrasts with bootstrap intervals, candidate-pool diagnostics, and human-calibration breakdowns.
- Extended results: The slot-sensitivity gap concentrates in NORMBODY, SCOPE, and TRIGGER rather than evidence or procedure.This localizes the strongest sensitivity to the rule’s semantic core.
B.1 Metric Sensitivity
Metric-sensitivity analysis places the disclosed 20% HARD-OK weighting in a stable region where GNRS-Search remains strongest, while a 40% weighting overemphasizes feasibility. The analysis also clarifies that HARD-OK is a diagnostic rather than a complete definition of rule quality.
- Metric sensitivity: 20% HARD-OK weighting yields GNRS-Search’s disclosed executable composite, while 10–30% weights preserve its strongest ranking across reported settings.GNRS-Search ranks first in the main benchmark, cross-model robustness columns, and held-out OOD evaluation for 10%, 20%, and 30% weights.
- Metric sensitivity: 40% HARD-OK weighting lets No LLM candidates overtake GNRS-Search in the GPT-4o-mini main setting.The paper therefore treats 40% as too feasibility-heavy for the headline metric.
- Metric sensitivity: GNRS-Search is strongest on pure Avg in the main and OOD settings before executable feasibility receives additional weight.In some cross-model settings, stronger generators make Direct Prompting competitive on pure Avg, but even a small feasibility weight favors GNRS-Search.
- Interpretation: The 40% failure case shows that HARD-OK is a diagnostic feasibility term rather than a complete definition of rule quality.No LLM candidates’ conservative feasibility can dominate at high weighting despite lower rubric quality.
- Scope boundary: Cross-model claims cover completed generator columns only because GPT-4o and GLM-4.6 were omitted after incomplete or timed-out runs.The completed run set contains 4224/4224 expected runs.
- Interpretation: The HARD-OK gap reflects richer selected payloads and more non-goal evidence payloads in full GNRS-Search, making its ratio-based grounding gate more fragile.GNRS-Search averages 16.65 LLM-augmented candidates per run, with ρ = 0.507, 6.19 directly supported payloads, and 6.08 inferred payloads.
B.4 Additional Result Details
Additional analyses show that GNRS-Search’s advantages persist across benchmark comparisons and evaluation checks, while revealing a quality–feasibility trade-off and bounded external validity.
- Bootstrap analysis: GNRS-Search’s main-benchmark OVERALL margin over No LLM candidates is positive with a 95% interval of [3.93, 7.53], while HARD-OK favors No LLM candidates.The combined score therefore distinguishes task validity from standalone binary feasibility.
- Ablation analysis: The no-search ablation fails on faithfulness and feasibility despite using the same broad rule inventory, indicating that candidate scoring and structured search contribute materially.The ablation retains the inventory and renderer but removes candidate scoring or search.
- Slot sensitivity: Slot-intervention audits achieved full replacement coverage, and the smaller evidence sensitivity is interpreted structurally rather than as an artifact of missing replacement candidates.Scope, trigger, and norm-body changes more directly alter who must act, what conduct is required, and when the rule activates; the paper presents this as a benchmark-specific diagnostic.
- Validity and limitations: Human calibration and judge-family checks support the automatic evaluation, while the benchmark scope excludes full institutional-policy diversity and complete-document interpretation.GNRS-Bench is controlled, and RealCharter-Bench uses normalized policy cards rather than complete source documents.
- RealCharter-Bench: On RealCharter-Bench, GNRS-Search leads Self-Refine on OVERALL, Avg, HARD-OK, and Faithfulness under the normalized task-card protocol.The comparison includes Self-Refine as the strongest LLM workflow baseline and hides the original source clauses from generators.