PaperlayerView exact revision on arXiv

Source-linked AI summary

Policy Central: Detecting Conflicts, Redundancy, and Similarity in Deployed Police Policy

Holly Baker, Michael Kuo, Kyana Pollas

arXiv:2610.29338v1cs.CLcs.CY

TL;DR

Police policies can contain contradictory or duplicative rules, and finding relationships by hand is slow. Policy Central compares a retrieval-based system with an exhaustive screen-and-confirm cascade; models judge handed pairs better than they find them. The system reports candidate pairs for human review, not policy decisions.

  • Problem

    Police policies can contain internal tensions, while finding related rules by hand is slow and errors have real field consequences.

    Source: §1 · p. 1

  • Method

    Policy Central classifies section pairs as Conflict, Redundancy, Similarity, or Different using Retrieve-Judge for library-wide scanning and Screen-Confirm for selected-document audits.

    Source: §1 · p. 1

  • Results

    Given pairs, judges reach 0.73 to 0.89, but recall falls when systems must find the pairs themselves.

    Source: §4.3 · pp. 5–6

  • Takeaways & Limitations

    Policy Central provides reviewers with candidate pairs to check; it does not replace their review, and its scans can be rerun after edits.

    Source: §5 · p. 6

    Absolute recall is optimistic because design changes were evaluated on the same 302 pairs without a held-out split.

    Source: Development and evaluation share one… · p. 7

Abstract

from arXiv · show

Policy Central is a deployed system that helps police departments find relationships between sections of their written policy: whether two sections conflict, are redundant, are merely similar, or are different. We describe the system and compare two architectures. Retrieve-Judge, a retrieve-then-classify pipeline, runs as a cheap library-wide health signal; Screen-Confirm, a screen-then-confirm cascade, is a selective audit of the documents a reviewer chooses, checking them exhaustively against one another. On a 302-pair benchmark drawn from one department's ~84 real policy documents, given each pair Screen-Confirm reaches 0.795 aggregate recall and 0.885 on Conflict, so current models judge well. Finding the pair is the hard part: Retrieve-Judge's composed recall falls to 0.457, and Screen-Confirm recovers 0.472 (0.615 on Conflict). Today these pairs are found by reading the whole library, nearly 20,000 sections, checked against one another and re-checked after every revision. A scan that returns half of them in minutes, at low cost, and can be re-run on demand changes the economics of policy review even before recall improves.

1 Introduction

Police policy can contain conflicting or duplicative rules, and finding such relationships by hand is slow and consequential. Policy Central surfaces four relationships through two architectures designed for different review needs.

1 Introduction

  • Policy Central is piloted at one U.S. police department and automatically surfaces relationships between policy sections.The system is described as having a broader rollout underway.Source: §1 · p. 1
  • It labels pairs Conflict, Redundancy, Similarity, or Different; Conflict is high-value but infrequent, while most library pairs are Different.Conflict means the sections cannot both be followed; Redundancy means one section is unnecessary because the other says the same thing.Source: §1 · p. 1
  • Retrieve-Judge is a retrieve-then-classify library-wide health signal, whereas Screen-Confirm audits documents selected by a reviewer.The two architectures serve distinct operational needs.Source: §1 · p. 1
  • The paper compares both architectures and two baselines on a benchmark of 302 real policy pairs, evaluating pair judgment and pair finding.It also reports the production cost and time of both operating modes.Source: §1 · p. 1

2 Related Work

Prior work addresses legal conflict, compliance, and text relations through varied methods, but the cited systems do not judge whether two sections within one policy library conflict. This paper frames policy review around both natural-language inference and the reviewer’s next action.

The problem

  • A review of over 11,000 pages of use-of-force policies found wide variation, while a systematic review of NLP in policing covered no policy-document work.The policies sometimes spread force rules across multiple sections of general orders; the NLP review is Sarzaeim et al. (2023).Source: §2 · p. 1

Prior systems

  • Prior approaches include extracted or hand-encoded knowledge graphs, generated or statutory-conflict benchmarks, compliance checks, duplication measurement, and NLI over defence policies.The cited procurement and maintenance systems check against external standards or certified sources, rather than judging conflict between sections of one library (Jo, 2026).Source: §2 · p. 2

Framing

  • Similarity-based retrieval can miss task-relevant conflicts because conflicting rules need not read alike; even GPT-4 is unreliable on context-dependent self-contradictions, and current LLMs fail to detect conflicts across clinical guidelines.Source: §2 · p. 2

3 System Design

Policy Central uses one ingestion pipeline to support two report-generating architectures. The system adapts relation labels to reviewer actions, then uses either similarity-based retrieval or an exhaustive screen-and-confirm cascade.

3 System Design

  • A reviewer selects a policy document, and the system reports related sections elsewhere in the library using Retrieve-Judge or Screen-Confirm.Retrieve-Judge retrieves nearest neighbours; Screen-Confirm checks every section of another document against every requirement of the selected document.Source: §3 · pp. 2–3

3.1 Relation Taxonomy

  • The four labels adapt NLI to reviewer actions: contradiction maps to Conflict, bidirectional entailment to Redundancy, and neutral to Similarity or Different.Similarity versus Different depends on whether a reviewer revising one section would need to read the other; Conflict is symmetric, unlike NLI’s directional framing. Table 1 pairs the four labels with real excerpt examples.Source: §3.1 · p. 3, Table 1 · p. 3

3.2 Data Ingestion

  • One pipeline creates a structural index, a distilled provisions store, and a vector index; provisions are extracted once per document and grounded to source chunks.The roughly 600-token chunks preserve source text, while a separate ingest-time LLM rewrites chunks as standalone requirement sentences; missed extractions are invisible on the anchor side.Source: §3.2 · p. 3

3.3 Retrieve-Judge

  • Retrieve-Judge embeds each leaf chunk, retrieves top-k neighbours, and judges each pair; similarity sampling can miss conflicts whose rules share no wording.The deployed audit uses k=20 and a 0.75 similarity threshold; the paper therefore treats it as a library-health scan rather than an audit.Source: §3.3 · p. 4

3.4 Screen-Confirm

  • Screen-Confirm cheaply screens every compared-document section against distilled anchor provisions, then confirms escalated pairs against the raw section.Any screen label other than Different escalates a pair; a confirm downgrade to Different drops it. A scan uses N screen calls plus M confirm calls, with distillation paid once per document.Source: §3.4 · p. 4

4 Evaluation

The 302-pair evaluation shows that judging supplied pairs works better than finding them, while the two architectures serve different deployment needs. Retrieve-Judge offers an inexpensive library-wide pass, whereas Screen-Confirm is used for targeted comparisons.

4.1 Benchmark and Its Construction

  • The benchmark contains 302 real-section pairs—26 Conflict, 57 Redundancy, 44 Similarity, and 175 Different—with semi-automatic labels; the mining pipeline is not evaluated as a detector to avoid circularity.People checked disputed rows after model-based labeling and review.Source: §4.1 · p. 4

4.2 Systems and Methodology

  • The evaluation compares finding a pair with judging a supplied pair across both architectures and two baselines; Retrieve-Judge estimates end-to-end recall by multiplying retrieval and judge recall, assuming independence.Screen-Confirm measures end-to-end recall directly through reach, escalation, and confirmation stages.Source: §4.2 · pp. 4–5

4.3 Results by Relation Class

  • 0.795 given-pair recall for Screen-Confirm falls to 0.472 end-to-end, while Retrieve-Judge drops from 0.654 to 0.457; Conflict end-to-end recall is 0.615 versus 0.418.The table distinguishes systems handed the pair from systems that must find it.Source: Table 3 · p. 5, §4.3 · pp. 5–6

4.4 Time and Cost on the Department’s Library

  • A library-wide Retrieve-Judge pass covers roughly 4,800 neighbour pairs for $13 in five minutes; a targeted Screen-Confirm scan of one SOP against the library scales to about an hour and $40–$80.The pilot’s main use for Screen-Confirm was comparing a document with a few documents under revision or needing alignment.Source: §4.4 · p. 6

5 Conclusion and Future Work

Policy Central gives reviewers candidate policy pairs rather than replacing manual review, but finding those pairs remains the main challenge. Its two scan modes serve different review workflows, and deployment work will focus on improving finding quality and expanding the pilot.

5 Conclusion and Future Work

  • Finding contradictory or repeated rules remains a manual task, and Policy Central supplies candidate pairs for reviewers rather than replacing their review.The exhaustive cascade trades precision for recall and can report on a document pair in minutes, allowing a reviewer to fix and re-check in one sitting.Source: §5 · p. 6
  • Table 4 reports production-scan cost and wall-clock time on the deployment with ten scan workers, counting screen-plus-confirm calls for Screen-Confirm and judge calls for Retrieve-Judge.Its projected row scales measured per-section cost and time and is an upper bound, not the intended use.Source: Table 4 · p. 6
  • Deployment priorities are improving finding quality from pilot feedback, extending rollout to further departments, and reporting usage data.The parser is pattern-based and tuned to one department’s directive formats, making ingestion the main barrier to other corpora.Source: §5 · p. 6

Limitations

The benchmark and evaluation have important scope and measurement limits, including semi-automatic labels, no held-out split, and unresolved parsing and retrieval effects. These constrain how confidently absolute recall and cross-architecture comparisons can be interpreted.

Limitations

  • The 302 real benchmark pairs have semi-automatic rather than subject-matter-expert labels, and its 26 Conflicts come from one department’s corpus, leaving Conflict estimates with wide intervals and transfer untested.The label set also lacks a “partly redundant” class, forcing partial-overlap pairs into Redundancy or Similarity.Source: Limitations · p. 7

Development and evaluation share one bench-

  • The definition and prompt changes were measured on the same 302 pairs without a held-out split, making absolute recall optimistic and change direction more trustworthy than magnitude.Model swaps also use prompts tuned against Sonnet 4.5, so they measure prompt transfer as well as model capability.Source: Development and evaluation share one… · p. 7
  • Retrieve-Judge judges leaf chunks while Screen-Confirm judges whole sections, leaving unit size uncontrolled; silent parsing boundary errors can alter what confirmation reads.The miss attribution checks text presence and section location, not boundary correctness, and no boundary-level parser accuracy is reported.Source: Development and evaluation share one… · p. 7
  • Screen-Confirm escalates 36–71% of sections depending on anchor size, but the measured gain over a single pass is recall, not cost per correct finding.Distillation recall is a lower bound under a token-overlap test, and the misses observed mostly include descriptive statements that the extraction prompt does not treat as rules.Source: Development and evaluation share one… · p. 7
  • Benchmark mining and production both use Haiku, so requirements it reliably fails to extract may be missing from both the benchmark and provisions store.Candidates were paired by shared terms, leaving no zero-lexical-overlap pair; enrichment for rare classes also means deployment precision will be lower.Source: Development and evaluation share one… · p. 7
  • Retrieve-Judge’s corpus-dependent neighbor sets can change after edits, causing fixed pairs to disappear and issue counts to stay flat or rise despite improvement.Screen-Confirm instead rechecks the full per-section space on each run, so its reported issues drop reliably as issues are fixed.Source: Development and evaluation share one… · pp. 7–8
  • Retrieve-Judge’s composed recall estimate assumes retrieval and judge difficulty are independent, applies retrieval measured on 137 mapped pairs to whole-class judge recall, and uses recall at 10 rather than the production threshold.The estimate is an upper bound on production retrieval; an end-to-end Retrieve-Judge measurement remains a next step.Source: Development and evaluation share one… · p. 8

Ethical Considerations

Policy Central is an advisory document-review tool: human agency personnel retain responsibility for deciding whether policies conflict and what to do. Its benchmark labels and Conflict recall do not support treating its output as authoritative or complete.

Ethical Considerations

  • The system proposes candidate relationships for human policy reviewers and never edits, approves, or enforces policy.Every finding includes model-attributed quotes for checking against source text.Source: Ethical Considerations · p. 8
  • Conflict findings are low-precision by design (0.404), and the production path recovers only 0.615 of benchmark Conflicts, so a clean report does not certify a clean library.The system is presented as a complement to review, not a certification.Source: Ethical Considerations · p. 8
  • Benchmark labels were semi-automatic rather than subject-matter-expert judgments, so they are not authoritative readings of policy.Source: Ethical Considerations · p. 8
  • The system is scoped to document review, not adjudicating individual conduct; proposed fixes or enforcement-relevant uses warrant a fresh risk assessment.Source: Ethical Considerations · p. 8

A Benchmark Construction and Example Provenance

The benchmark’s 302 real policy pairs were assembled through mechanical candidate mining, followed by model-assisted labeling and targeted human verification. Candidate generation used three routes, and the source passages remain verbatim document text.

A Benchmark Construction and Example Provenance

B Judge-Model Leaderboard

Across eight judge models, Retrieve-Judge aggregate recall clusters between 0.46 and 0.57, with model choice moving results less than the architecture. A Llama-specific prompt improves judge-only Conflict recall while precision remains near 0.49.

B Judge-Model Leaderboard

  • Every judge model lands between 0.46 and 0.57 aggregate recall, a spread of about 0.12, so model choice moves results less than architecture.The shared retrieval factor sets the absolute level of composed scores, not the model ordering.Source: B Judge-Model Leaderboard · p. 10, Table 5 · p. 10
  • Shared leaf-corrected retrieval recall was 0.571 for Conflict, 0.846 for Redundancy, and 0.480 for Similarity in composed rows.The table combines measured per-class judge recall with the same retrieval recalls across those rows.Source: Table 5 · p. 10
  • A from-scratch Llama-specific prompt raised judge-only Conflict recall from 0.654 to 0.808 while Conflict precision held near 0.49.Under the Sonnet-tuned prompt, Llama 3.3 70B had judge-only Conflict recall of 0.654.Source: B Judge-Model Leaderboard · p. 10

C Negative Ablations

Several tested retrieval, prompting, and context strategies produced no usable improvement and were excluded from the deployed system. Their failures ranged from unchanged retrieval scores to lower Conflict precision or higher cost.

C Negative Ablations

  • Five embedding-side retrieval fixes did not improve Conflict or Similarity R@10, and two regressed it.The tests varied parent indexing, generated context, titles, embedding dimension, and quantization; the gap was attributed to chunk boundaries and the leaf filter.Source: C Negative Ablations · p. 11
  • Cohere reranking changed no composed cell, while HyDE found the right topic but not the annotated partner and gave no net R@10 gain.Reranking moved per-class R@10 by at most ±0.05.Source: C Negative Ablations · p. 11
  • A forced chain-of-thought protocol lowered Retrieve-Judge Conflict precision from 0.444 to 0.304 by applying its rule inconsistently across near-duplicate phrasings.Source: C Negative Ablations · p. 11
  • Whole-document judging on an 18-pair pilot scored within noise of quote-only judging at roughly 39× the input tokens per pair.Documents reached up to 533K tokens.Source: C Negative Ablations · p. 11
  • Agentic context fetching was a wash on 302 pairs—24 rows fixed and 24 broken—at about 3.7× cost, and sometimes turned correct Different calls into false Conflicts.Source: C Negative Ablations · p. 11

D Which Fixes Moved Recall

On the production index, Screen-Confirm’s end-to-end recall was 0.472 overall and 0.615 on Conflict. Stage-by-stage diagnosis showed how ingestion and design fixes raised recall, with precision costs.

D Which Fixes Moved Recall

  • 0.472 aggregate recall and 0.615 Conflict recall were achieved by Screen-Confirm on the production index.Table 7 tracks where the 127 gold positives ended, distinguishing absent text, missing provisions, screen losses, confirm losses, and correctly found pairs.Source: D Which Fixes Moved Recall · p. 11, Table 7 · p. 12
  • Cross-document pairs reached 0.541 recall, versus 0.377 for within-document pairs.Source: D Which Fixes Moved Recall · pp. 11–12
  • Fixing three ingest faults raised recall from 0.260 to 0.354; provision windows raised it to 0.449, and passage-level confirmation to 0.472.The ingest faults were overwritten split-PDF chunks, dropped short chunks, and truncated distillation batches; the later fixes also increased false positives among Different pairs from 12 to 20 of 175.Source: D Which Fixes Moved Recall · p. 12

E Benchmark-Scale Cost and Latency

This section documents benchmark-scale evaluation and methodological details for the two pipelines, including cost and latency, per-class precision, deployment caveats, prompts, confirmation rules, and zero-shot baselines.

E Benchmark-Scale Cost and Latency

  • Table 8 reports benchmark-run cost and latency for 302 pairs, using a sequential single worker; its figures are not production costs.Costs are summed from logged token counts at list rate; Retrieve-Judge output tokens were estimated from a 40-pair sample, and wall-clock times come from an earlier 376-pair run.Source: Table 8 · p. 12, E Benchmark-Scale Cost and Latency · p. 12

F Precision and Confidence Intervals

  • 1.000 Redundancy precision and 0.612 Similarity precision; the cascade’s errors were Redundancy pairs called Similarity, with no pairs mislabeled Conflict or Different.Table 9 reports Screen-Confirm precision and recall per class, with 95% Wilson intervals on recall.Source: F Precision and Confidence Intervals · p. 12, Table 9 · p. 12
  • 23 of 26 benchmark Conflicts were caught (recall 0.885), but precision falls from 0.404 to about 0.06 if Conflicts are ten times rarer.In the 302-pair benchmark, the cascade flagged 57 Conflicts: 23 correct and 34 false flags; under the stated rarity assumption, the false flags remain while correct flags fall to about 2.Source: F Precision and Confidence Intervals · p. 12

G Prompts

  • The prompt appendix reproduces five prompts verbatim, with Retrieve-Judge’s judge prompt longest and the distillation and screen prompts shorter.The longer judge prompt carries label definitions, decision order, and five worked examples; shorter prompts rely on the provisions list for context.Source: G Prompts · p. 12

G.1 Provision Distillation Prompt (Ingest, Fast Model)

  • Provision distillation extracts each distinct requirement as a standalone sentence prefixed by its exact source-chunk header.The fast-model system prompt receives batches of up to 100 chunks, each prefixed with its header.Source: §G.1 · pp. 12–13

G.2 Retrieve-Judge Judge Prompt

  • Retrieve-Judge’s judge applies content-based label definitions in a fixed order, checking Redundancy before Conflict, Different, and then Similarity.Its rules distinguish material normative differences from wording changes and require a same-officer, same-situation contradiction for Conflict.Source: §G.2 · p. 13

G.3 Screen-Confirm Screen Prompt (Fast Model)

  • The fast-model screen flags every provision that plausibly relates to the Policy B section and includes uncertain cases for later re-checking.It labels Conflict, Redundancy, or Similarity and skips only completely unrelated provisions.Source: §G.3 · pp. 14–15

G.4 Screen-Confirm Confirm Prompt (Strong Model)

  • The strong-model confirmer classifies passages by their requirements first and consults the screen’s label only as a tie-breaker when genuinely undecided.For end-to-end runs, it first isolates the single passage in the Policy B section that addresses the same subject as Policy A.Source: §G.4 · p. 15

G.5 Screen-Confirm Confirm Prompt, Same-Document Variant

  • Screen-Confirm treats explicit exceptions and differing scope as compatible, reserves Conflict for free-standing opposite rules, and labels compatible same-subject obligations with material normative differences Similarity rather than Redundancy.Redundancy requires the same obligation and differs only in non-normative ways; a different actor or threshold is a material delta.Source: §G.5 · p. 16
  • The confirmation step uses the first-pass label only to break a genuine tie and returns JSON with confirmation status, final label, rationale, and relevant passage quotes.The prompt says not to switch labels merely because the first-pass guess was provided; quote fields may be empty for Different.Source: §G.5 · p. 16

G.6 Baseline Setups

  • Llama zero-shot classifies each pair in one quote-only call at temperature 0 with no examples, choosing among Conflict, Redundancy, Similarity, and Different.The baseline uses us.meta.llama3-3-70b-instruct-v1:0 via Bedrock.Source: §G.6 · p. 16
  • DeBERTa NLI scores each label for the concatenated pair using a zero-shot hypothesis template, selecting the highest entailment score and breaking ties in label order.The model is MoritzLaurer/deberta-v3-large-zeroshot-v2.0 and is not fine-tuned.Source: §G.6 · p. 16
Loading 2610.29338v1…