Source-linked AI summary
Natural-Language Policies to Executable Decisions: An Interpretable Large Language Model Framework
Ziqiang Zhang, Jing Ma, Zilong Wang, Jiayuan Chen, Yi Qiao, Yu He, Wei Zhang, Dai Cheng, Xiaoyu Shen
TL;DR
Tourism pricing automation must reconcile highly informal travel orders and evolving, open-ended policies with strict reliability and auditability requirements. The paper presents a production framework that restricts LLMs to structured extraction and bounded policy selection, compiles policies into interpretable condition trees, and performs numeric pricing deterministically. In six months, the deployed system processed 3,960 orders while reducing the order-management team from 15–20 to 3 and handling time from ∼10 minutes to < 2 minutes.
Problem
Tourism pricing must process heterogeneous, often ambiguous travel orders and open-ended policies, while unconstrained LLMs lack the stability and interpretability needed for high-stakes pricing.
Method
LLMs perform structured extraction and discrete policy/path selection, while policies become interpretable condition trees and all numeric calculations run in a deterministic engine.
Results
3,960 orders were processed in six months, reducing the order-management team from 15–20 to 3 and per-order handling time from ∼10 minutes to < 2 minutes.
Takeaways & Limitations
The framework supports auditable, human-governed pricing and is designed for transfer to other policy-driven domains with evolving natural-language rules.
Takeaways & Limitations
End-to-end accuracy remains difficult to evaluate for ambiguous orders, and empirical validation is limited to a single tourism deployment site.
Abstract
from arXiv · showhide
Pricing automation in large-scale tourism is challenging because travel orders are highly unstructured, while pricing policies are complex, rapidly evolving, and inherently open-ended. Traditional rule engines are brittle and costly to maintain, whereas unconstrained LLM agents lack the reliability and auditability required for financial decisions. We present a production-grade LLM-powered pricing system with a strict decision boundary: LLMs perform structured extraction and bounded policy/path selection, while all numeric pricing, including total-price computation, is executed deterministically. Policies are compiled into interpretable condition trees, enabling open-ended support for new clauses and evolving rules without code changes, while exposing auditable artifacts for human-in-the-loop control. Periodic fine-tuning on logged traces further improves tree induction and path matching. Deployed at a municipal state-owned tourism enterprise across 7 scenic sites and 12 business categories with 1,500+ operators and 1,000+ active policies, the system processed 3,960 orders in six months, reduced the order management team from 15-20 to 3, and cut per-order handling time from 10 minutes to <2 minutes.
1 Introduction
Tourism pricing automation must handle highly unstructured travel orders and open-ended, evolving policies while remaining reliable, interpretable, and auditable. The proposed system confines LLMs to bounded reasoning tasks and uses auditable artifacts and human oversight to support production pricing.
- Travel orders range from digital forms to messages and handwritten notices, with critical details often implicit or ambiguous.Examples include informal scenic-spot abbreviations and dates inferred from context.
- Pricing policies contain interacting conditions and novel clauses that traditional manually encoded rule engines cannot readily anticipate.Conditions include seasonal windows, group-size thresholds, and bespoke contractual clauses.
- Unconstrained LLMs offer linguistic flexibility but lack the stability, interpretability, and inspectability required for high-stakes pricing.These limitations hinder non-technical users’ ability to review or correct intermediate reasoning.
- The system uses LLMs to match order facts with policy condition trees, supporting arbitrary ad-hoc rules and seasonal exceptions without code changes or retraining.Policy logic is organized into explicit trees that preserve natural-language semantics and can be reviewed by tourism managers.
- 3,960 orders were processed in six months while the order-management team fell from 15–20 to 3 and handling time dropped from ∼10 minutes to < 2 minutes.The deployment also exposed OrderFacts, condition trees, and logged decision traces for bounded human confirmation and overrides.
- The framework targets policy-driven industries where open-ended logic, informal inputs, strict correctness, and human-centered governance coexist.The paper identifies insurance underwriting, compliance checking, and contract execution as related domains.
2 Background
Tourism pricing follows policy onboarding, order intake and quotation, and settlement verification, but both policy logic and incoming orders are difficult to standardize. These constraints require systems that decouple policy logic from fixed schemas and expose interpretable intermediate artifacts for human intervention.
- Pricing operations comprise policy onboarding, order intake and quotation, and settlement and verification.Managers publish policies, coordinators match orders to active policies, and calculated prices are checked against source policies.
- Pricing conditions span resources and products, with virtually unlimited combinations of age brackets, group thresholds, seasonal overlaps, and contractual clauses.New policies can introduce entirely novel logic.
- 97% of orders were instant-messaging screenshots, and nearly half required manual item-list edits before quotation.Orders combine dates, destinations, and casual notes without a fixed template, while initial itineraries often differ from executed plans.
- Traditional rule engines require coordinated changes to database schemas, rule logic, and input interfaces whenever new policy attributes or predicates appear.This hard-coding cycle makes adaptation costly and inflexible.
- A production system must decouple policy logic from software schemas and externalize reasoning into interpretable artifacts that preserve natural-language semantics.Human intervention is treated as a first-class operation rather than an exception.
3 Methodology
The methodology confines LLMs to structured extraction and bounded discrete selection, while deterministic components perform routing, validation, and all numeric price computation. Policies become interpretable condition trees, and editable artifacts plus human overrides support governed execution.
- System Design: LLMs perform structured extraction and discrete decision selection, while a deterministic engine performs all numeric pricing calculations.This strict decision boundary is intended to eliminate numeric hallucination and make uncertainty explicit for human verification.
- Human-in-the-Loop Governance: The system exposes policy structures, order facts, routing metadata, and decision steps through operator interfaces supporting confirmation and override across the reasoning chain.The architecture is decomposed into policy onboarding stages A0–A3 and price calculation stages B0–D, with tiered L1–L3 intervention.
- Policy Onboarding and Tree Induction: Policy documents are transformed offline into condition trees whose internal nodes encode applicability conditions and whose root-to-leaf paths each specify one price.The onboarding pipeline includes text extraction, catalog grounding, node extraction, and validation before publication.
- Policy Onboarding and Tree Induction: The resource catalog defines recognized atomic resources and feasible product bundles, while duplicate bundle definitions are rejected through exact resource matching.This keeps catalog data consistent and constrains downstream grounding and execution.
- Policy Onboarding and Tree Induction: Induced trees store metadata, natural-language conditions, and leaf-level price specifications, with numeric values retained as static fields rather than calculated by the LLM.The production-faithful structure uses an outer metadata layer and an arbitrarily nested condition tree.
- Policy Onboarding and Tree Induction: Every tree must pass validators for schema completeness, units, numeric sanity, catalog resolution, constraint format, and cross-path conflicts before publication.Failures block execution and route the tree to operator review through an editable onboarding console.
- Order Parsing and Price Calculation: Incoming orders are parsed into editable OrderFacts, enumerated into feasible bundles, deterministically filtered by policy metadata, and matched by bounded LLM path selection before deterministic execution.Operators can correct ambiguous fields and reconcile resources before routing; the final PriceSpec is returned at the selected leaf.
4 Experiment
The evaluation uses production tourism orders to measure workload, decision accuracy, and operational impact across a staged decision-boundary funnel. Results show substantial automation gains while retaining operator intervention for mismatches and errors.
- Deployment and data: 3,960 production orders were evaluated across seven scenic areas, twelve tourism categories, and more than 1,500 operational staff.The dataset covers 2025H2 logs, including 3,842 screenshot orders and 118 Word uploads.
- Deployment and data: 97.0% of the 3,960 orders were instant-messaging screenshots processed through OCR, motivating evaluation of informal-input handling.Over 70% involved multiple resources and 25% contained bundle products.
- Decision-boundary evaluation: 47.9% of screenshot orders required resource edits, while 2.8% required bundle or product switching during the decision-boundary funnel.These interventions address mismatches between extracted or stated resources and the executed itinerary or product composition.
- Decision-boundary evaluation: 5.7% of orders requiring no composition-level intervention still needed L2 tree switching, indicating that metadata-based routing resolved most policy-selection decisions automatically.This result is reported among 1,892 orders requiring no composition-level intervention.
- Correctness results: Zero fatal numeric errors occurred across 2,598 product instances, while unit-price accuracy reached 85.2% on 1,349 information-consistent orders.The remaining 14.8% were L1 path-selection errors recoverable through a single UI click.
- Operational impact: Policy onboarding decreased from approximately twenty minutes per policy with developer involvement to roughly one minute without developer involvement.Average order processing time was about three minutes across OCR extraction, pricing execution, and operator confirmation.
5 Discussion
The system confines LLMs to discrete selection tasks and executes numeric pricing deterministically, using condition trees as both executable policy representations and operator interfaces. Discussion highlights zero fatal numeric errors, targeted correction through interpretable boundaries, and scope limitations in path selection and validation.
- Decision boundary: LLMs perform resource grounding, condition extraction, and path selection, while validated leaf specifications supply deterministic numeric computation.This boundary prevents the model from generating prices directly.
- Decision boundary: Zero fatal numeric errors across 2,598 product instances demonstrate the production benefit of separating LLM decisions from numeric execution.The reported error definition includes invalid units, aggregation mismatches, and unsupported prices.
- Interpretability and governance: Condition trees function as the primary editing interface for non-technical operators, with a 5.7% L2 override rate after tree construction.Logged disagreements can be attributed to specific decision boundaries for targeted iteration.
- Error analysis: The 14.8% unit-price mismatch reflects under-specified eligibility evidence and OCR noise in screenshot orders rather than a single failure source.Some discounted clauses require evidence unavailable in OrderFacts, while garbled screenshots can produce incorrect extraction.
- Generalization: Condition-tree induction, deterministic candidate enumeration, and evidence-backed discrete selection are presented as domain-agnostic components for other policy-driven industries.Examples include insurance underwriting, regulatory compliance, and contract execution.
6 Conclusion
The paper concludes with a production framework that converts natural-language pricing policies into interpretable condition trees and processes informal orders through deterministic pricing. A six-month deployment processed 3,960 orders and reported operational gains, supporting structured, auditable LLM integration in high-stakes workflows.
- Framework: The framework converts natural-language pricing policies into executable, interpretable condition trees and informal travel orders into deterministic pricing decisions.Its central separation places language-model work in structured extraction and bounded selection rather than numeric price generation.
- Deployment outcome: 3,960 orders were processed during a six-month deployment at a municipal, state-owned tourism enterprise.The deployment reported substantial operational gains and used structured, auditable reasoning interfaces.
- Conclusion: The deployment demonstrates that LLMs can be integrated into high-stakes decision workflows when paired with structured and auditable reasoning interfaces.The stated generalization target is policy-driven domains where interpretability, reliability, and nonexpert usability matter.
Limitations
The deployment has limitations in policy-tree maintenance, resource consistency, evaluation coverage, and empirical generalizability. These constraints leave practical opportunities for automated tree management, improved order clarification, broader annotations, and multi-domain validation.
- Policy maintenance: Manual grouping merges policies with similar customer scope and validity patterns, but accumulated policies may increase matching ambiguity and operator burden.Automatic tree merging and splitting are identified as future work.
- Order consistency: 47.9% of production orders required L3-a resource edits because stated resources differed from the itinerary actually executed.Richer order formats or multi-turn clarification could reduce this operator workload.
- Evaluation scope: Track 2 correctness metrics cover only 1,349 information-consistent orders with composition and tree selection preresolved.End-to-end accuracy on ambiguous full-order inputs remains difficult to evaluate without additional ground-truth annotations.
- Generalizability: All experiments were conducted at a single tourism deployment site, so validation in domains such as insurance or regulatory compliance is still needed.The architecture is designed to be domain-agnostic, but cross-domain empirical support is not yet established.