PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy-as-Skill: Governed LLM Decision Support with Evidence, Deterministic Control, and Audit

Kabeh Mohsenzadegan, Vahid Tavakkoli, Kyandoghere Kyamakya

arXiv:2609.27087v1cs.AIcs.SC

TL;DR

Policy and compliance decision support needs evidence validation, human review, version control, and auditability beyond fluent answers or retrieval alone. Policy-as-Skill packages these functions as an executable, versioned runtime with separable retrieval, validation, and optional deterministic control. The evaluation supports selective controller use: broad control can reduce accuracy on some task families, while governance and review properties remain distinct from the underlying model decision.

  • Problem

    Policy Q&A and decision support need evidence, review routing, version control, and auditability, while retrieval and prompts alone do not specify these runtime controls.

    Source: §I · p. 1

  • Method

    Policy-as-Skill packages policy as a reusable, versioned executable capability with explicit evidence, decision, review, audit, failure, prompt, and context elements.

    Source: §I · p. 1, §III · p. 2

  • Results

    PaS contributes stronger review routing and complete native audit reconstruction, while blanket deterministic control can reduce accuracy on risk classification and policy QA.

    Source: §VI · p. 5, §VII · p. 6

  • Takeaways & Limitations

    Use deterministic controls selectively where policy semantics are explicit, while preserving evidence and uncertainty and keeping humans involved for ambiguous or high-impact cases.

    Source: §VI · p. 5, §VII · p. 6

    The 600-task evaluation informed system/controller development, so it is not a frozen system-level held-out test.

    Source: A. Benchmark, Corpus, and Provenance · p. 3

Abstract

from arXiv · show

Organizations increasingly use LLMs for policy, compliance, risk, and operational decision support, requiring evidence validation, review routing, version control, and auditability. We introduce Policy-as-Skill (PaS), a modular runtime that packages these functions as executable, versioned policy capabilities. Thirteen methods are evaluated with a fixed Gemma4 backend on 600 development tasks. PaS+Audit achieves 53.8% exact accuracy, macro-F1 0.346, review F1 0.854, citation precision 1.000, policy-reference recall 0.984, and audit completeness 1.000, outperforming LLM+RAG on most governance and review metrics. Deterministic control raises aggregate accuracy to 61.2% but is strongly task dependent, supporting selective rather than universal rule-based intervention.

I. INTRODUCTION

Policy decision support requires more than fluent answers or retrieved policy text: it must handle evidence, review, versioning, and audit. Policy-as-Skill packages policy as an executable, versioned capability and separates retrieval, control, and audit functions.

I. INTRODUCTION

  • LLM policy support must identify applicable sources, distinguish requirements from guidance, route uncertain or high-impact cases to humans, and preserve versioned evidence for reconstruction.The paper situates these needs in policy Q&A, compliance triage, risk classification, and operational decision support.Source: §I · p. 1
  • RAG and Policy-as-Prompt do not specify policy-version selection, missing-evidence failure, mandatory review, or the fields required for an auditable record.The paper also notes concerns about natural-language instructions remaining stable across changing contexts,.Source: §I · p. 1
  • Policy-as-Skill packages policy as a reusable, versioned, executable capability rather than only as text, with retrieval, audit validation, and deterministic control separable.The controller can be enabled where policy semantics are sufficiently explicit; it can improve some task types and degrade others.Source: §I · p. 1
  • The paper contributes a formal skill contract, reproducible ablations, a 13-method evaluation over 7,800 task-method runs, sensitivity analyses, and a distinction between model-level unseen data and system-level held-out evaluation.The implementation and experiment artifacts are openly available.Source: §I · p. 1

II. RELATED WORK

Prior work applies retrieval, prompting, and reasoning or action techniques to policy-facing tasks, but retrieval and prompts alone do not provide the runtime controls needed for governed decisions. PaS frames organizational policy as a versioned executable contract that includes evidence, validation, escalation, failure, and audit requirements.

II. RELATED WORK

  • Policy-facing LLM studies demonstrate extraction and reasoning over policy text, but evidence retrieval alone does not specify runtime precedence, failure behavior, review routing, or reconstructable audit records.RAG supplies non-parametric evidence for knowledge-intensive NLP.Source: §II · p. 1
  • Policy-as-Prompt work treats policy as model instruction, [6], while related studies examine prompt governance and framing,, [12],.PaS does not equate a natural-language system prompt with enforceable control; it places the prompt within a versioned executable contract.Source: §II · pp. 1–2
  • Reasoning-and-action approaches include chain-of-thought, ReAct, multi-agent prompting, and code-as-policies, but generally target task execution rather than organizational-policy accountability.The passage describes these approaches as informing the decomposition of generation, action, and control.Source: §II · p. 2
  • Prompt optimization and action-policy research includes learned prompt strategies [18]– and control or recommendation policies –[27], distinct from PaS’s governance-policy object.PaS policy objects represent organizational or regulatory constraints to retrieve, validate, audit, and selectively enforce.Source: §II · p. 2
  • AI governance frameworks emphasize oversight, traceability, accountability, risk management, and lifecycle change control –; PaS binds these concerns to each policy capability.The bound elements include policy scope, evidence requirements, review triggers, failure rules, and audit fields.Source: §II · p. 2

III. POLICY-AS-SKILL ARCHITECTURE

The architecture represents a policy skill as a formal, executable contract with explicit identity, scope, evidence, decision, review, audit, failure, prompt, and context elements. Its runtime keeps retrieval and audit validation separable from optional deterministic control.

III. POLICY-AS-SKILL ARCHITECTURE

  • A policy skill is defined as a tuple whose identity, version, retrieval scope, required evidence, decision schema, review triggers, audit fields, failure behavior, prompt, and context are explicit.Context may include data category, user role, processing location, or responsible owner.Source: §III · p. 2
  • The registry implements skills for compliance checking, risk classification, conflict detection, evidence-grounded recommendation, review routing, and policy-version adaptation.The passage also states that the runner validates materialization of all ten tuple components in PaS traces.Source: §III · p. 2
  • The architecture’s formal skill tuple is mapped to executable runtime artifacts.Table I is titled “Formal Skill Tuple and Executable Realization.”Source: Table I · p. 2
  • The runtime produces a candidate answer and structured fields, while validation and audit can run independently and the deterministic controller remains optional.The research runner validates that all ten tuple components are materialized in PaS traces.Source: §III · p. 2
  • Four variants separate skill selection and scoped retrieval, deterministic control, audit validation without control, and the combination of control and audit.The controller inspects retrieved normative statements and task context, not task IDs, expected labels, or expected answers.Source: §III · p. 2

IV. EVALUATION DESIGN

The evaluation uses 600 cases across four task families and an 11-fixture policy corpus. Its labels and system-development history constrain what the benchmark can establish.

IV. EVALUATION DESIGN

  • The benchmark contains 600 instances, balanced across four task families, with 453 cases requiring human review and all 150 conflict-detection cases labeled needs review.Because every conflict case is an escalation case, that subset tests conflict recognition and routing but is less discriminative as a multi-class benchmark.Source: A. Benchmark, Corpus, and Provenance · p. 2
  • The 11 policy fixtures are synthetic, illustrative, or real-world-inspired rather than authoritative policies, and labels are single-expert-curated rather than an independently adjudicated legal gold standard.The EU AI Act sample is a simplified educational artifact, not the regulation itself.Source: A. Benchmark, Corpus, and Provenance · p. 2
  • The reported run uses seed 7 and a fixed gemma4:e2b backend, with no benchmark data used for model training or fine-tuning.Earlier trace-level analysis of the 600 tasks informed system/controller development, so this is not a frozen system-level held-out test.Source: A. Benchmark, Corpus, and Provenance · pp. 2–3

B. Methods and Research Questions

The study compares thirteen methods using the same configured model backend and asks how they perform on decision correctness, review routing, evidence and audit properties, controller effects, and composite-score sensitivity. Its metrics separate substantive decisions from audit metadata and treat the weighted readiness index as an engineering diagnostic.

B. Methods and Research Questions

  • Thirteen methods are compared, including four PaS variants, with all model-based methods using the same configured backend.The research questions cover decision correctness and review routing, PaS evidence and audit properties, controller effects, and readiness-score sensitivity.Source: B. Methods and Research Questions · p. 3

C. Metrics and Fairness

  • Exact categorical accuracy and macro-F1 measure decision quality, while review precision, recall, and F1 measure human-review routing.Citation precision and policy-reference recall assess evidence grounding and recovery of expected policy references.Source: C. Metrics and Fairness · p. 3
  • Common trace completeness is 1.0 for every method, while native audit completeness separately measures whether a method emits reconstruction fields.Primary task accuracy does not reward audit metadata.Source: C. Metrics and Fairness · p. 3
  • The weighted governance-readiness index is secondary because its weights are design choices; 286 nonnegative weight combinations are evaluated.The combinations lie on a 0.1 simplex over decision, evidence, governance, and answer-similarity components.Source: C. Metrics and Fairness · p. 3

V. RESULTS

PaS+Audit provides a strong balanced profile, while PaS Full has the highest aggregate exact accuracy but lower macro-F1. The aggregate comparison needs task-level context, including the escalation-oriented conflict-detection labels.

A. Primary Decision and Review Performance

  • 0.612 exact accuracy makes PaS Full the aggregate leader, but its macro-F1 is 0.306 versus PaS+Audit’s 0.346; PaS+Audit records review F1 0.854.PaS+Audit has 0.538 exact accuracy; LLM+RAG, the strongest conventional baseline, has 0.498 exact accuracy and review F1 0.695.Source: A. Primary Decision and Review… · p. 3, Figure 2 · p. 4
  • Fig. 3 shows that adding the controller to audit-only PaS changes outcomes in different directions across task types.Fig. 2 presents exact decision accuracy and human-review F1 for representative methods; PaS audit/retrieval variants provide the strongest review routing.Source: Figure 3 · p. 4, Figure 2 · p. 4
  • The aggregate comparison requires task-type context because every conflict-detection reference label is needs review.Source: A. Primary Decision and Review… · p. 4

B. Controller Intervention Analysis

The controller improves aggregate accuracy relative to PaS+Audit, but interventions have mixed task-level effects. The results support selective use where policy semantics are well specified.

B. Controller Intervention Analysis

  • 44 tasks, or 7.3 percentage points overall, are the net improvement after PaS Full changes decisions on 401 of 600 tasks.Of 401 interventions, 181 correct incorrect decisions, 137 turn correct decisions incorrect, and 83 change one incorrect class to another.Source: B. Controller Intervention Analysis · p. 4
  • +59.3 points on conflict detection contrasts with −23.3 points on risk classification and −8.7 points on policy QA; compliance changes +2.0 points.The risk-classification degradation is statistically clear in this development set; the QA and compliance changes are not significant at 0.05.Source: B. Controller Intervention Analysis · p. 4, Figure 4 · p. 4
  • Across the 450 tasks excluding conflict detection, PaS Full reaches 48.2% exact accuracy versus LLM+RAG’s 50.4%, while PaS+Audit reaches 58.2%.The corresponding differences versus LLM+RAG are −2.2 points for PaS Full and +7.8 points for PaS+Audit.Source: B. Controller Intervention Analysis · p. 4
  • The controller should be applied selectively where policy semantics are well specified, rather than treated as a universal correctness layer.Source: B. Controller Intervention Analysis · p. 4

C. Evidence, Audit, and Composite Sensitivity

Citation validity, decision correctness, audit reconstruction, and composite readiness capture distinct properties. Composite rankings vary with engineering weights, and the reported latency timings are descriptive rather than controlled performance evidence.

C. Evidence, Audit, and Composite Sensitivity

  • PaS+Audit and PaS Full reach citation precision 1.000 and policy-reference recall 0.984, while Keyword Search has exact decision accuracy 0.285 despite high evidence faithfulness.Evidence overlap and decision correctness should be reported separately.Source: C. Evidence, Audit, and Composite… · p. 4
  • PaS+Audit and PaS Full score native audit completeness 1.000, versus approximately 0.812 for RAG/PaP variants and 0.625 for closed-book LLM variants.All methods have common trace-envelope completeness 1.000; native audit completeness reflects reconstructability, not decision accuracy.Source: C. Evidence, Audit, and Composite… · p. 4
  • Across 286 weightings, PaS Full ranks first 35.1% of the time, PaS+Audit 30.8%, and Keyword Search 33.9%, making the composite unsuitable as the primary correctness claim.The governance-readiness index is 0.714 for PaS Full and 0.636 for LLM+RAG.Source: C. Evidence, Audit, and Composite… · p. 4
  • Mean latency is 9.72 s for PaS+Audit and 16.65 s for LLM+RAG, but timings are descriptive rather than controlled edge-device or performance-attribution evidence.The run used Docker/WSL2 and an external CPU host; run order and caching were not designed for performance attribution.Source: C. Evidence, Audit, and Composite… · p. 4
  • Table II reports primary substantive metrics—exact accuracy, macro-F1, and review F1—separately from citation precision, policy-reference recall, and native audit completeness.Table III reports the controller effect as PaS Full minus PaS+Audit exact accuracy, with paired bootstrap confidence intervals.Source: Table II · p. 5, Table III · p. 5

VI. DISCUSSION

The discussion frames PaS as explicit, lifecycle-managed governance around model decisions, while arguing that deterministic control should be selective and benchmark conclusions remain bounded by system-development-informed evaluation.

VI. DISCUSSION

  • PaS separates retrieval, audit, and governance properties from model decisions; the strongest conventional baseline already retrieves and cites policy evidence well, while PaS strengthens review routing and native audit reconstruction.These governance properties can matter even when the underlying model decision is unchanged.Source: §VI · p. 5
  • Deterministic control improves a narrowly specified escalation task but degrades two other task families, so controllers fit rules with high semantic precision rather than ambiguous classification or interpretive QA.For ambiguous cases, preserve model uncertainty and rely more on evidence, validation, and human review.Source: §VI · p. 5
  • The LLM was not trained or fine-tuned on the benchmark, but system-level error analysis influenced controller development, so the 600 tasks are not a pristine held-out test of the final system.This distinction matters when important system behavior resides outside model weights.Source: §VI · p. 5

A. Operational Implications for Decision Support

  • A policy skill binds retrieval scope, evidence requirements, decision schema, review triggers, audit fields, failure behavior, and contextual boundaries to an explicit version.When policy changes, historical traces retain the version, evidence, and routing logic used for earlier decisions, supporting replay or regression analysis.Source: A. Operational Implications for… · p. 5
  • Evidence-grounded assistance, governed assistance with validation, and deterministic intervention are distinct operating profiles, and no single profile is universally preferable.Treat controller eligibility as a property of each skill; explicit prohibitions, mandatory evidence, and unambiguous escalation suit deterministic enforcement better than interpretive QA or classification-heavy policies.Source: A. Operational Implications for… · p. 5
  • Human review is a governed outcome: missing evidence, unresolved conflicts, or high-impact contexts can trigger visible escalation instead of forcing a model decision.The repository includes provenance, frozen-test manifests, overlap checks, human-annotation support, and physical-device measurement scripts for future validation, but these do not make the present benchmark held out.Source: A. Operational Implications for… · p. 5
  • Exact decision quality, evidence grounding, review routing, and audit completeness answer different operational questions; composite rankings change with engineering weights.The discussion recommends retaining a small metric portfolio and defining acceptable trade-offs per use case rather than treating one readiness score as universal.Source: A. Operational Implications for… · p. 6

VII. CONCLUSION

The conclusion presents PaS as explicit, executable governance for LLM decision support, not as a way to make the base model intrinsically more intelligent. It favors selective deterministic controls and calls for frozen, independently validated evaluation before system-generalization or legal-correctness claims.

VII. CONCLUSION

  • PaS makes policy scope, evidence requirements, review triggers, failure behavior, validation, and audit reconstruction explicit and executable without claiming to improve the base LLM's intrinsic intelligence.Source: §VII · p. 6
  • Skill-scoped retrieval and audit validation support review routing, citation validity, and reconstructable records without necessarily overriding model decisions.Blanket controller use can reduce accuracy on risk classification and policy QA, despite improving narrowly specified escalation behavior.Source: §VII · p. 6
  • A frozen, independently validated held-out evaluation remains necessary before making system-generalization or legal-correctness claims.Source: §VII · p. 6
Loading 2609.27087v1…