Source-linked AI summary
SCOPE: Compiling Natural-Language Policies into Neurosymbolic Predicate Programs
Arnon Mazza, Elad Levi
TL;DR
Compound policies over long transcripts and agent trajectories are difficult for single-model classifiers to enforce accurately and consistently. SCOPE compiles each policy into an executable program of neural predicates, extractors, and composition code, optimized with adversarial counterexamples. Across policy-enforcement tasks, it achieves the highest accuracy in most settings and transfers to other executors without re-optimization.
Problem
Compound policies combine conditions, exceptions, counts, and interactions across long inputs, while existing classifier approaches face accuracy, stability, or cost limits.
Source: §1 · pp. 1–2Method
SCOPE compiles a policy into neural predicates, deterministic extractors, and composition code, jointly optimizing the program and an adversarially grown training pool.
Source: §1 · pp. 2–3Results
Across four policy-enforcement tasks, SCOPE achieves the highest accuracy in most settings, and programs transfer to other executors without further optimization.
Source: §1 · p. 3, §6 · p. 10Takeaways & Limitations
Programs run with weak executors can reach accuracy comparable to frontier reasoning models while producing stable, predicate-traceable verdicts.
Source: §1 · p. 3, §6 · p. 10
Abstract
from arXiv · showhide
Policy enforcement for LLM agents has shifted from single-turn content moderation to compound, application-specific policies evaluated over multi-turn transcripts and tool-call trajectories. Fine-tuned guardrail models are efficient but degrade on compound policies, and while LLM-as-a-judge approaches achieve higher accuracy, they incur high inference cost and tend to produce unstable predictions. Neurosymbolic methods offer a natural alternative, decomposing the policy into narrow neural decisions composed under explicit logic. However, existing symbolic layers lack the required expressiveness and are optimized against fixed data rather than the policy itself. To address this, we introduce SCOPE, which compiles a policy into an executable program of neural predicates executed by small models, deterministic extractors, and code that composes their outputs. The program is optimized jointly with its training data, using adversarial generation of counterexamples that target where the program deviates from the policy. We evaluate SCOPE on a benchmark of real-world policy tasks against prompt optimization, task decomposition, and workflow generation methods, achieving the highest average accuracy across tasks and models. Programs optimized with the weakest executor transfer to other models without further optimization. This allows them to be deployed in production on inexpensive models, where they reach accuracy comparable to frontier reasoning models. Unlike these monolithic models, SCOPE produces substantially more stable predictions, and each verdict is interpretable through the predicate outputs that produced it. Additionally, we validate the key components of our method through an ablation study. SCOPE thus enables reliable, interpretable, and efficient policy enforcement at production scale.
1 Introduction
SCOPE compiles compound natural-language policies into executable predicate programs and jointly optimizes the program and its training data with adversarial counterexamples. Across four policy-enforcement tasks, it achieves the best accuracy in most task-executor settings while supporting weak-executor deployment and traceable verdicts.
1 Introduction
- Compound policies over transcripts, plans, and agent trajectories combine conditions, exceptions, counts, and interactions, making overlooked clauses consequential.The unit of judgment has shifted beyond a single message or flat category.Source: §1 · p. 1
- Fine-tuned guardrails lose accuracy on compound policies, while LLM-as-a-judge predictions are unstable and costly at production scale.Even reasoning models can collapse beyond a complexity threshold.Source: §1 · p. 2
- SCOPE compiles policies into neural predicates, deterministic extractors, and code, then uses adversarial counterexamples to revise both predicate wording and composition.The adversary targets local predicate drift and systemic failures in the composition as a whole.Source: §1 · pp. 2–3, Figure 1 · p. 2
- SCOPE attains the best accuracy in most task-executor settings, transfers from its weakest executor to other models, and produces stable, traceable verdicts.Ablations identify adversarial generation as central to the gains.Source: §1 · p. 3
2 Related Work
Prior policy-enforcement methods trade expressiveness against cost and consistency, while symbolic program approaches often restrict the formalism or leave programs fixed. SCOPE combines executable composition with joint policy-directed optimization and adversarially grown training data.
2 Related Work
- Existing enforcers trade expressiveness against cost and consistency: compact models struggle with compound rules, while frontier judges are costly and inconsistent.Blocklists and regular expressions are repeatable and inexpensive but limited to literal patterns.Source: §2 · p. 3
- Restricted symbolic formalisms cannot naturally express conditional checks, runtime-selected spans, and repeated checks that real policies may require.Code-based approaches gain expressiveness, but prior examples may leave programs fixed or predicates frozen.Source: §2 · p. 4
- SCOPE combines executable Python composition with joint optimization of program structure and predicates against the policy itself.This connects expressiveness with policy-directed fitting rather than a static benchmark or proxy.Source: §2 · pp. 3–4
- SCOPE grows its training data adversarially during decomposition search, using a strong-model policy verifier rather than a sound solver.This makes counterexamples component-specific but sacrifices formal guarantees for policies outside solver capabilities.Source: §2 · p. 4
3 Methods
SCOPE searches for an executable predicate program that reproduces policy verdicts under a weak executor, starting from a small labeled seed pool. Its optimizer revises the program using traced failures while adversaries add policy-checked counterexamples targeting predicate and composition errors.
3 Methods
- SCOPE starts from a small human-labeled seed pool and alternates program revision with adversarially generated samples labeled by a teacher applying the full policy.The search stops at a score threshold or round limit and returns the program scoring highest on the fixed seed pool.Source: §3 · pp. 4–5
3.1 The Decomposition Program
- A decomposition program combines fixed-label natural-language predicates, deterministic extractors, and Python code that maps predicate outputs to a verdict.The code can select spans, apply predicates conditionally or repeatedly, and compute counts and thresholds exactly.Source: §3.1 · p. 5
3.2 Optimizer
- The optimizer uses predicate traces to localize errors, revises only in ways grounded in policy clauses, and backtracks when recent rounds fail to improve.Because training pools grow, it selects the returned program by its score on the fixed seed pool.Source: §3.2 · pp. 5–6
3.3 Adversarial Generators
- Local and systemic adversaries generate counterexamples for predicate-level drift and composition-level omissions or misrouting, respectively.A teacher labels each sample without seeing its intended label; a sample enters the pool only when the labels agree.Source: §3.3 · p. 6
4 Experimental Setup
The experiments evaluate SCOPE on four policy-enforcement tasks against prompt optimization, task-decomposition, and workflow-generation baselines. With gpt-5.4-nano, SCOPE outperforms the prompt-optimization and decomposition baselines on every task.
4.1 Datasets
- The four tasks cover critic auditing, repeated-intent escalation, research-plan validation, and identifying dashboard tiles referred to by a user.Tile-match uses a sample-specific label space and mean Jaccard scoring; its test set includes 55 no-tile hard negatives.Source: §4.1 · pp. 6–7
4.2 Baseline methods
- GEPA and MIPROv2 rewrite a fixed task description without decomposing it; MIPROv2 also selects few-shot demonstrations.Both are optimized on SCOPE’s initial training pool, use the same executor, and are scored on the same held-out test set.Source: §4.2 · p. 7
- CheckEval decomposes policies into binary checklist questions, while RLIE induces natural-language rules from labeled samples; both are limited to binary-label tasks.Figure 2 compares test accuracy by executor and method, with a panel per task; CheckEval and RLIE appear only for Escalation and Plan.Source: §4.2 · p. 7, Figure 2 · p. 7
- AFlow searches open-ended executable workflows and was restricted to its Custom operator, though its instructions could elicit intermediate reasoning.The comparison left that capability in place because it is inherent to AFlow.Source: §4.2 · pp. 7–8
- SCOPE outperforms all prompt-optimization and decomposition baselines on every task with gpt-5.4-nano.Table 2 reports test accuracy, except mean Jaccard for Tile-match.Source: Table 2 · p. 8
4.3 Implementation Details
- The decomposition process uses Claude Opus 4.8 in an agentic loop, while executor models return predicate labels without rationales.Search runs for at most eight iterations, stopping early when training-pool accuracy reaches 0.98; adversarial samples are capped to avoid dominating the human-labeled pool.Source: §4.3 · p. 8
5 Results and Analysis
SCOPE attains the highest accuracy or ties for it in most task-executor settings, with a smaller accuracy gain from stronger executors than prompt optimization. Its decompositions transfer to other models, and adversarial sample generation improves accuracy on three tasks.
5 Results and Analysis
- SCOPE achieves the highest accuracy or ties for it in 10 of 12 task-executor cells; AFlow leads in the other two.With gpt-5.4-nano, SCOPE's standard deviation is at most 0.03 on every task.Source: §5 · p. 8
- 0.09 vs. 0.26 average accuracy gain separates SCOPE from prompt optimization when moving from the weakest to strongest executor.Figure 3a shows the gain is smaller for SCOPE.Source: Figure 3 · p. 9, §5 · p. 9
5.1 Cross-Executor Transferability
- Decompositions found with gpt-5.4-nano improved accuracy over undecomposed runs on five open-weight models without further optimization or reasoning.Average gains were roughly 0.11 on Coaching, Escalation, and Plan, and 0.33 on Tile-match.Source: Table 3 · p. 9, §5.1 · p. 9
5 Results and Analysis
- Weaker executors yield more predicates, with the largest increase from gpt-5.4-mini to gpt-5.4-nano.When an executor cannot answer a predicate accurately, the search splits it further.Source: §5 · p. 9
- SCOPE retains the best-scoring candidate from a search initialized with the undecomposed task, so one predicate remains a possible outcome.On Coaching with gemini-3.5-flash, the search converged to one predicate and accuracy rose from 0.87 to 0.93.Source: §5 · p. 9
5.2 Ablation: Removing Adversarial Samples
- Removing adversarial sample generation reduced accuracy on Coaching from 0.839 to 0.783, Plan from 0.865 to 0.813, and Tile-match from 0.782 to 0.744.The ablation found no effect on Escalation.Source: Table 4 · p. 10, §5.2 · p. 10
5.1 Cross-Executor Transferability
- DeepSeek-V4.1-Flash and Qwen3-235B-A22B reached accuracy comparable to high-end reasoning models on undecomposed tasks.Both transferred-model runs used reasoning disabled.Source: §5.1 · pp. 9–10
- On Tile-match, only 5–7% of samples changed prediction across five runs for DeepSeek-V4.1-Flash and Qwen3-235B-A22B, versus 37% for gpt-5.5 and 26% for gemini-3.1-pro.Each prediction is traceable to explicit predicates.Source: §5.1 · p. 10
6 Conclusion
SCOPE compiles natural-language policies into executable predicate programs and optimizes them with an adversarially expanded training pool. Across four tasks, it performs best in most settings, transfers across executors, and yields stable, traceable verdicts, though correctness is not formally verified.
6 Conclusion
- Across four policy-enforcement tasks, SCOPE achieves the highest accuracy in most settings, with its largest gains under the weakest executor.Programs transfer to other executors without further optimization and reach accuracy comparable to frontier reasoning models when run by non-reasoning open-weight models.Source: §6 · p. 10
- SCOPE treats a natural-language policy as a specification to compile into neural predicates, deterministic extractors, and composition code.The program is optimized jointly with an adversarially grown training pool.Source: §6 · p. 10
- Its decompositions are less sensitive to executor choice, and its verdicts are substantially more stable and traceable to predicate outputs.The conclusion reports accuracy comparable to frontier reasoning models at a fraction of the cost when non-reasoning open-weight models run the programs.Source: §6 · p. 10
- Program correctness rests on a strong teacher’s reading of the policy rather than a formal verifier, and verification remains open.The authors suggest the co-evolving program-and-counterexample optimization may extend to rubric-based LLM evaluation and reward modeling.Source: §6 · p. 10
A Task decomposition example
The Coaching example uses a two-phase program to detect policy violations and audit an evaluator’s verdict and evidence, ending in one of three outcomes.
A Task decomposition example
- The Coaching program first detects message violations with five semantic predicates and two deterministic word- and sentence-count checks.Its inputs include the coaching message, system state, and the evaluator’s verdict and reasoning.Source: A Task decomposition example · p. 18
- A second phase compares detected violations with the evaluator’s verdict and checks whether each cited violation is supported by the message.Two predicates audit the evaluator’s evidence on the message and each evidence item independently.Source: A Task decomposition example · p. 18
- The decision tree terminates in Pass, Fail, or Bug.Source: A Task decomposition example · p. 18
B Dataset Details
The dataset appendix details sources, construction, splits, and test-label distributions for four policy-enforcement tasks.
B Dataset Details
- Coaching adapts real-world recommendation-system cases into three classes, with 103 training and 71 test samples; its test set has 29 Bug, 24 Fail, and 18 Pass labels.Source: B Dataset Details · p. 19
- Escalation extends DynaGuard with generalized human-agent escalation, sharper intent-based repetition, and added human-annotated cases; its splits contain 85 training and 286 test samples.The test set has 147 True and 139 False samples, distributed across repetition and escalation conditions.Source: B Dataset Details · p. 19
- Plan adds authored invalid plans and boundary-valid examples to research-agent traces, with 82 training and 81 test samples.The test set contains 41 Fail and 40 Pass labels.Source: B Dataset Details · p. 19
- Tile-match is a multi-label task with a sample-specific candidate-tile set, using 39 training and 124 test samples.The test set includes 45 multi-tile, 24 single-tile, and 55 no-match cases, the latter designed as hard negatives.Source: B Dataset Details · p. 19
C Cost Analysis
The cost analysis compares inexpensive-executor decompositions with frontier-model inference on undecomposed tasks and reports lower costs for the decompositions.
C Cost Analysis
- 9 to 47 times less: decomposition costs less than applying gemini-3.1-pro to the undecomposed tasks.Source: C Cost Analysis · p. 19
- The decomposition estimate assumes $0.15 per 1M input tokens and negligible output cost because it runs without reasoning.Source: C Cost Analysis · p. 19
- The comparison uses gemini-3.1-pro with dynamic thinking, the cheaper of the two frontier models by list price.The passage gives list prices of $2 and $12 per 1M input and output tokens for gemini-3.1-pro, versus $5 and $30 for gpt-5.5.Source: C Cost Analysis · p. 19
D Detailed Results by Executor Model
The detailed-results table reports test accuracy across four tasks and three executor models, while the adversarial examples include systemic and predicate-local failures in intermediate decompositions.
D Detailed Results by Executor Model
- Table 6 covers four classification tasks and three executor models, with accuracy measured by task-appropriate metrics.Coaching, Escalation, and Plan use percentage correct; Tile-match uses mean Jaccard index.Source: Table 6 · p. 20
- The reported values average nine measurements: three optimization runs and three test-evaluation repetitions.Numbers are rounded to two decimal places.Source: Table 6 · p. 20
E Adversarial examples
- The adversarial examples comprise a systemic composition-program gap and a local predicate error, both targeting intermediate decompositions.Source: E Adversarial examples · p. 21
E.1 Systemic adversarial examples
The systemic example audits whether a fitness-message critic’s verdict and cited reasoning are correct. It exposes a deterministic branch that accepts a Fail verdict despite fabricated evidence.
E.1 Systemic adversarial examples
- The audit assigns each coaching-message sample exactly one label: Pass, Fail, or Bug.Pass and Fail require both the underlying message status and the evaluator’s judgment to be correct; Bug covers evaluator errors.Source: §E.1 · p. 21
- The procedure checks the message against rubric dimensions, compares that result with the evaluator’s verdict, then validates the cited reasons for Fail cases.A mismatched verdict or an invalid dimension/evidence entry makes the sample a Bug.Source: §E.1 · p. 22
- The deterministic sentence-count branch accepts a Fail label when the message violates the rule, even if the evaluator’s cited evidence is fabricated.In the example, the message has two sentences, but the cited evidence claims three statements joined by semicolons; the rule requires right-verdict/wrong-evidence cases to be Bugs.Source: §E.1 · p. 25
E.2 Predicate-local adversarial examples
The escalation task compares whether escalation was warranted with whether the agent connected the user to a human. Its example shows how new information can change whether repeated requests count toward escalation.
E.2 Predicate-local adversarial examples
- A repeated unresolved intent counts toward escalation unless a turn adds a new constraint, distinct question, or information that changes the situation.Escalation is warranted when the same unresolved intent is raised at least three times.Source: §E.2 · pp. 25–26
- The final classification is True when the agent’s escalation action matches whether escalation was warranted, and False when the two do not match.Source: §E.2 · pp. 25, 27
- In the blender example, the delivery-status update changes the case from a lost parcel to disputed delivery, so the turn does not count as repetition and escalation is not warranted.The decomposition instead treats the update as a repetition by narrowing “changes the situation” to changes in the user’s fundamental need or underlying problem.Source: §E.2 · p. 27
F Task decomposition prompts
These task-decomposition prompts use the original rule to guide how predicates, helpers, and code reconstruct its decisions, diagnose failures, and propose rule-grounded fixes while protecting correct cases. They also direct a DECIDER to select and record program edits for later backtracking.
F Task decomposition prompts
- The failure analyzer traces misclassifications through predicate calls, input slices, helpers, and combination logic, using nearby correct cases as a contrast set to check proposed fixes.It distinguishes incorrect predicate judgments from structural or slicing bugs and recommends targeted changes that avoid flipping correct rows.Source: F Task decomposition prompts · p. 28
- Every diagnosis and recommendation must follow from the original rule, not patch specific sample wording; the analyzer reports failure patterns and corrective moves without prescribing a complete rewrite.Recommendations target predicates, helpers, or combination code according to the root cause, and at least one corrective move is required when failures exist.Source: F Task decomposition prompts · p. 29
- The decomposition assigns subjective judgments to fixed-label classifier predicates, deterministic parsing and slicing to helpers, and all deterministic composition to code that returns the final label.Predicates are pure and may classify multiple slices; helpers can return values or slices, while combination code controls calls, loops, and final decisions.Source: F Task decomposition prompts · p. 30
- Each predicate must be a standalone instruction carrying its own question, label meanings, scope, definitions, examples, and exceptions from the original rule.Repeated sections may be evaluated in a loop when evidence indicates that per-section calls help, but a whole-sample call is the default; semantic matching should use classifiers rather than brittle substring rules.Source: F Task decomposition prompts · p. 31
- The DECIDER may follow or override per-predicate recommendations, choosing to keep, reword, or split predicates and updating combination_code when needed.Rewording changes only a predicate’s decision wording; splitting creates finer predicates and requires combination_code to consume their outputs.Source: F Task decomposition prompts · pp. 32–33
- Within combination_code, predict(name, text) applies a predicate to text, while helpers slice the sample input before it is passed to predict.The sample is the input-block text described by sample_schema, and declared helpers are callable in scope.Source: F Task decomposition prompts · p. 33
- The DECIDER must list every change from the base decomposition in applied_moves, with its target, move, and rationale.The record captures the search direction so later backtracking can avoid repeating it.Source: F Task decomposition prompts · p. 33
Related papers
This paper cites
- $τ$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains2024
- PAL: Program-aided Language Models2022
- Large Language Models as Optimizers2023
- Decomposed Prompting: A Modular Approach for Solving Complex Tasks2022
- Neural Module Networks2015
- NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails2023