PaperlayerView exact revision on arXiv

Source-linked AI summary

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

Kshitij Mishra, Abhijith Sharma, Nils Lukas, Salem Lahlou

arXiv:2610.22124v1cs.AI

TL;DR

Capability and content-safety benchmarks do not directly test whether tool-using agents follow policies during otherwise legitimate tasks, and multi-turn failures are hard to cover with scripted cases. PROBE trains realistic, multi-turn auditors that score full traces; in SKYASSIST, trained auditors expanded policy discovery, including when smaller than the target.

  • Problem

    Capability and content-safety evaluations do not directly measure procedural policy compliance, while multi-turn traces create a large space that static scripted suites cannot cover.

    Source: §1 · pp. 1–2

  • Method

    PROBE trains an auditor to interact through the user-facing dialogue as a plausible customer or tester and rewards distinct high-severity failures while shaping realistic behavior and tool-state progress.

    Source: §1 · p. 2, §4.2 · p. 5

  • Results

    60% held-out policy coverage, versus 35% for a strong zero-shot auditor, was achieved in SKYASSIST, with multi-policy failures in up to 94% of evaluation episodes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Policy compliance is a distinct evaluation axis that calls for adaptive behavioral auditors to uncover reachable failures in realistic interaction traces.

    Source: §7 · p. 10

    The study covers two environments and two target families, and the target rarely reaches booking-level policies; deterministic predicates may also miss semantic violations.

    Source: §7 · p. 10

Abstract

from arXiv · show

Modern agentic systems extend language models with external tools, enabling them to complete complex tasks through multi-turn interactions. As these agents are deployed in consequential settings, they are often constrained by policies governing pricing, privacy, confirmation, safety, and other requirements. However, existing evaluation methods leave a critical gap: capability benchmarks measure what agents can accomplish, while content-safety red-teaming measures harmful generations, but neither tests whether agents comply with predefined policies during otherwise valid tool use. Such policy violations are difficult to audit because their effects may emerge only after several turns, span multiple tool calls, or surface in downstream systems. We propose PROBE (Policy Rule Observation via Behavioral Exploration), an algorithm for training LLM-based auditors that interact with deployed agents as realistic users and probe them for policy violations through multi-turn, tool-mediated dialogue. We formalize agent auditing as a two-player Markov game and propose a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity. To study training objectives and reward design, we compare GRPO, DPO, and GFlowNet-based optimization in a controlled setting. Across extensive experiments, auditors trained with PROBE uncover substantially broader and higher-yield policy violations than same-architecture zero-shot adversaries, even when the auditor is significantly smaller than the target agent. Our results show that multi-turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing capability and safety benchmarks. PROBE offers a scalable path toward automated policy auditing for real-world agentic systems and suggests a co-evolutionary loop in which auditors expose failures and agents improve from auditor feedback.

1 Introduction

Tool-using agents must follow procedural policies across full interaction traces, a property not directly measured by capability or content-safety benchmarks. PROBE trains smaller auditors to uncover reachable violations through realistic multi-turn interaction; in SKYASSIST, trained auditors broaden policy discovery despite being smaller than the target.

1 Introduction

  • Capability benchmarks test task completion and content-safety benchmarks test disallowed text, but neither directly measures compliance with prescribed policies during legitimate tasks.Policy evidence may depend on tool calls, arguments, confirmation timing, or outputs hidden from the final response.Source: §1 · pp. 1–2
  • Because multi-turn traces span user phrasing, database state, tool availability, and policy preconditions, PROBE trains adaptive auditors rather than relying only on scripted edge cases.The approach mirrors penetration testing by searching for failures reachable through the user-facing interface.Source: §1 · p. 2
  • PROBE trains a smaller model to act as a plausible customer or tester, scoring full traces while the frozen target's weights, prompts, database, and verifier labels remain hidden.The introduction frames the contribution as formalizing auditing as a two-player Markov game, training auditors with a curriculum, and rewarding distinct trace-level failures.Source: §1 · p. 2
  • 60% held-out policy coverage, up from 35% for a strong zero-shot auditor, was achieved by trained auditors in SKYASSIST, with multi-policy failures in up to 94% of episodes.Under deterministic re-scoring, auditors reached 55% coverage versus 5% for a static suite and transferred to τ-bench [Yao et al., 2024].Source: §1 · p. 2
  • The controlled study compares GRPO, DPO, and GFlowNet and reports that reward design interacts with optimizer choice; verifiers were also checked against human labels.The introduction notes stronger baselines and transfer evaluation on τ-bench [Yao et al., 2024].Source: §1 · p. 3

2 Related Work

PROBE addresses a gap between task-completion evaluation and red-teaming for harmful text: it adaptively searches tool interactions for procedural policy violations. Its objective is to train an external auditor against a frozen agent, rather than improve the deployed assistant.

2 Related Work

3 Policy-Compliance Auditing Game

The auditing game models an agent's policies and tools over a full execution trace, while the auditor interacts through public dialogue alone. Auditing therefore seeks high-severity, diverse failures through plausible user behavior under partial observation.

3 Policy-Compliance Auditing Game

  • A tool-augmented target is defined by its language-model policy, typed tool surface, and operational policies governing dialogue and tool traces.The policy set is formalized alongside the target agent as part of the game.Source: §3 · p. 3
  • Policies are paired with severity values and verifiers that evaluate the full trace of dialogue, target tool calls, outputs, and environment state.This trace-level framing distinguishes the game from token-level safety evaluation.Source: §3 · p. 3
  • The auditor observes public history and a customer scenario, but tool calls and state updates remain internal to the target-environment interaction.The auditor sees only the assistant's public response, making the game partially observed.Source: §3 · pp. 3–4
  • A useful auditor seeks high-severity failures, distinct policies rather than repeated easy triggers, and behavior realistic enough for a user or tester.The SKYASSIST overview situates this task within a tool-and-policy environment with verifiers and 20 policies.Source: §3 · p. 4, Figure 2 · p. 4

4 PROBE

PROBE combines a verifier-first flight-booking environment with rewards for distinct violations, progress, novelty, and plausible user behavior. A horizon curriculum supports multi-turn discovery, while GRPO, DPO, and GFlowNet supply different optimization biases.

4.1 Environment and Verifier-First Policy Specification

  • The SKYASSIST environment has four read-only and four mutation tools, with 20 policies spanning business, safety, privacy, and operational requirements.Sixteen policies use deterministic predicates; four use a hybrid LLM judge during training, followed by deterministic re-scoring after human validation.Source: §4.1 · p. 4

4.2 Reward as an Auditing Specification

  • The violation reward credits distinct severity-weighted policy failures, giving the first hit full credit, the second half credit, and later repeats none.Progress rewards entry into tool-relevant states, novelty favors undercovered policies, and plausibility penalizes assistant-like or fabricated user messages; progress and novelty weights are annealed in opposite directions during training.Source: §4.2 · p. 5

4.3 Curriculum and Rollout Generation

  • The horizon curriculum advances from 4 to 8, 12, and 14 turns when rolling violation yield exceeds 15% with patience 50.Each update samples four candidate messages from a shared prefix and rolls each against the frozen target for three continuation turns.Source: §4.3 · p. 5, Figure 3 · p. 5

4.4 Optimizer Instantiations

  • GRPO upweights high-return samples within each group and is mode-seeking, while DPO uses winner–loser preferences and is invariant to monotone reward reshaping.GFlowNet is mode-covering because trajectories are sampled in proportion to reward, but its benefit depends on calibrating rewards for policy diversity rather than total verifier mass.Source: §4.4 · pp. 5–6

5 Experiments

Experiments evaluate a fine-tuned auditor against a frozen, larger tool-using target under restricted access. The evaluation uses matched held-out scenarios and measures distinct policy discoveries alongside diagnostic metrics.

5 Experiments

  • The Qwen3-4B-Instruct auditor is fine-tuned with 4-bit NF4 QLoRA to probe a frozen Qwen3-30B-A3B target without access to its internals or verifier labels.The auditor has approximately 47M trainable parameters.Source: §5 · p. 6
  • 50 matched held-out scenario seeds, each evaluated for at most 8 public turns, instantiate valid, database-aware travel requests to avoid unreachable probes.Scenarios include valid routes, dates, passenger constraints, budgets, and modification or cancellation intents.Source: §5 · p. 6
  • Learned auditors are compared with random scenario sampling and an untrained Qwen3-30B zero-shot auditor, with methods sharing the target, verifiers, seeds, and turn budget.Additional static, few-shot, cooperative, and diversity-seeking baselines appear in Section 6.3.Source: §5 · p. 6
  • Suite-level policy coverage measures the share of 20 policies triggered anywhere in 50 episodes, prioritizing distinct discoveries over repeated hits.The paper also reports coverage mean and standard deviation over three independent held-out samples and bootstrap intervals for yield.Source: §5 · p. 6
  • Yield/100, B4/S5/B6 hit share, multi-policy rate, and entropy supplement coverage; yield can be inflated by repeated easy policies.Entropy is used only as a secondary concentration diagnostic.Source: §5 · p. 6

6 Results and Analysis

Training broadens policy discovery beyond zero-shot and scripted baselines, but coverage is constrained by the depth of tool states the target reaches. Reward calibration and per-policy credit shape optimizer behavior, while deterministic rescoring preserves the main ranking and transfer results show gains in additional settings.

6 Results and Analysis

  • Best trained 4B auditors found 12/20 policies versus 7/20 for zero-shot 30B, while reducing B4/S5/B6 hit share from 88% to 77% and raising entropy from 2.45 to above 3.1.The table evaluates 50 matched held-out scenarios using the training-time verifier suite.Source: §6 · p. 7, Table 1 · p. 7
  • The GRPO dedup-only ablation reached 1914 hits per 100 episodes and a 100% multi-policy rate but only 55% coverage; remaining gaps chiefly require deeper operational states.Undiscovered cases depend on mutation, lookup, or multi-leg states, while extra raw hits often revisit shallow turn-level policies.Source: §6 · p. 7

6.1 Reward and Optimizer Ablations

  • Customer-role prompting, scenario conditioning, progress, novelty, and plausibility improve realistic probes and coverage, while per-policy credit limits repeated-hit inflation and coverage remains stable across shaping weights.Replacing per-policy credit with generic diversity reward moves coverage for GFlowNet from 45% to 35%.Source: §6.1 · pp. 8–9
  • Calibration raises representative GRPO and GFlowNet training yields by 63% and 78%, while DPO changes less; GFlowNet's highest training yield does not deliver the highest held-out coverage.DPO reaches the same best held-out coverage as GRPO with fewer raw hits; GFlowNet reward-proportional sampling needs careful calibration.Source: §6.1 · pp. 8–9, §6 · p. 8

6.2 Error and Qualitative Analysis

  • Successful probes use plausible under-specified requests and stacked customer constraints, but shallow persistence can repeat B4/S5/B6 omissions without reaching new tool states.Target reluctance to call mutation tools limits observable coverage; one group-booking trace never reached book_flight, preventing the O1 verifier from firing.Source: §6.2 · p. 9

6.3 Verifier Validation and Stronger Baselines

  • Deterministic re-scoring removes S5, lowering best PROBE coverage from 60% to 55% without changing rankings; trained 4B auditors beat zero- and few-shot 30B baselines.The learned judge matched annotator consensus on only 39.6% of labeled instances, with 0% precision on S5.Source: §6.3 · pp. 9–10

6.4 Transfer to an Unseen Target Family and to τ-bench

  • Without adaptation, Qwen-trained PROBE-DPO exposes 8/20 policies (40%) on an unseen Gemma target, compared with 55% in-domain and 5% for the static suite.The transfer result is for the SKYASSIST setting.Source: §6.4 · p. 10, Table 3 · p. 10
  • On τ-bench, training raises coverage from 2/10 to 3/10 policies and violations from 13 to 29 per 50 episodes (+123%); a 4B auditor reaches similar coverage with only 7 violations.The benchmark uses 10 authored policies and the same Qwen3.5-9B base for both auditors.Source: §6.4 · p. 10, Table 3 · p. 10

7 Discussion: Toward Continuous Agent Auditing

PROBE frames policy compliance as a distinct evaluation axis and trains smaller auditors to expose reachable failures through realistic, multi-turn interactions. The authors position it as a complement to existing benchmarks and describe a continuous audit-and-retest loop, while noting limits in environments, targets, and policy reach.

7 Discussion: Toward Continuous Agent Auditing

  • PROBE treats policy violations as trace-level events and trains a smaller auditor to expose them through curriculum-based, multi-turn interaction.The authors conclude that realistic interaction traces require adaptive behavioral auditors to uncover reachable failures.Source: §7 · p. 10
  • A new domain can use the same training loop and reward with a tool environment, policy taxonomy, and verifiers; discovered traces can become regression tests or training data for re-auditing patched agents.PROBE is presented as a complement to task-completion and safety evaluations, not a replacement.Source: §7 · p. 10
  • The study covers two environments and two target families, and the auditor rarely reaches booking-level policies because over 90% of target tool calls are searches.Several of the highest-severity policies remain unreached; deterministic predicates may also miss semantic violations, and uncertainty uses held-out samples from one checkpoint per method.Source: §7 · p. 10

Appendix

  • The appendix covers the auditing game, optimizers, SKYASSIST, verifier validation, baselines, transfer, reward ablations, qualitative traces, and implementation details.Its sections are organized across Sections A–K.Source: Appendix · p. 14

A Policy-Compliance Evaluation Gap

Capability and content-safety evaluations do not establish whether an agent follows operational policies throughout tool use. Because relevant failures depend on multi-turn trace context and varied trigger conditions, adaptive probing complements fixed tests.

A Policy-Compliance Evaluation Gap

  • Operational policy compliance concerns whether the full execution trace obeys rules for tool use, disclosure, confirmation, privacy, and state preconditions.Capability measures task completion, while content safety concerns harmful or disallowed text; these are logically distinct criteria.Source: A Policy-Compliance Evaluation Gap · p. 14
  • An agent can complete a task and produce benign text yet violate policy through the timing of a tool call or the relation between dialogue, arguments, and preconditions.A successful flight booking before explicit itinerary and price confirmation illustrates the distinction.Source: A Policy-Compliance Evaluation Gap · pp. 14–15
  • Fixed scripted tests sample only a small part of the interaction space shaped by database states, user phrasing, tool availability, and multi-turn conditions.The same scripted suite reaches 5% coverage without an answer key and 55% when given each policy’s trigger in advance.Source: A Policy-Compliance Evaluation Gap · p. 15
  • PROBE combines learned adaptive probing with multi-turn tool use and trace-level policy verifiers, distinguishing it from approaches that target harmful responses or fixed trajectories.The comparison describes its combination of learned probing, adaptive interaction, diverse coverage pressure, and task-policy targets.Source: Table 4 · p. 15, A Policy-Compliance Evaluation Gap · p. 15
  • Raw verifier-hit maximization can repeatedly trigger easy omissions instead of exploring distinct policies, motivating severity-weighted deduplication, plausibility, progress, and novelty in the reward.Policies differ in firing semantics, including whether violations occur on many assistant turns or require rare tools or database states.Source: A Policy-Compliance Evaluation Gap · p. 15

B Formal Auditing Game and Evaluation Metrics

The formalization models auditing as partially observed interaction between a user-facing auditor, a frozen tool-using target, and verifiers with access to the full trace. It pairs trace-level policy scoring with coverage-oriented metrics and compares optimizer biases and the SKYASSIST test setting.

B Formal Auditing Game and Evaluation Metrics

  • The auditor interacts using public dialogue and a scenario, while the target uses its prompt and tools and verifiers inspect the private trace, making the game partially observed.The auditor lacks access to tool outputs, database state, target weights, and verifier labels; this asymmetry makes discovered failures reachable through ordinary user interaction.Source: B Formal Auditing Game and Evaluation… · pp. 15–16, Table 5 · p. 16
  • Verifiers score full traces with per-policy predicates that can evaluate events in context, including whether a required confirmation occurred earlier.Sixteen predicates are deterministic; four use a hybrid LLM judge during training, while deterministic re-scoring uses predicates for all twenty policies.Source: B Formal Auditing Game and Evaluation… · p. 16
  • Suite-level coverage is the headline campaign metric, while deduplicated and severity-weighted violations avoid letting repeated trivial firings dominate the audit score.Raw yield diagnoses concentration on high-rate policies; practitioners prioritizing remediation should also consult severity-weighted yield.Source: B Formal Auditing Game and Evaluation… · pp. 16–17
  • Plausibility is a constraint rather than an outcome, and tool-depth reach probabilities track whether auditors reach states needed for high-severity policies.Many such policies require mutation or multi-leg states, beyond query-level interaction.Source: B Formal Auditing Game and Evaluation… · p. 17
  • Coverage uncertainty is estimated across independent held-out samples because a within-run bootstrap can only remove discovered policies and understate spread.Evaluations use 50 held-out episodes; coverage moves in 5-point steps, so a one-step difference is treated as a tie unless yield supports it.Source: B Formal Auditing Game and Evaluation… · p. 17

C RL Algorithm Details

  • GRPO and DPO favor high-reward trajectories, whereas GFlowNet samples trajectories in proportion to reward and can continue sampling lower-reward violation modes.DPO uses only best-versus-worst candidate identities and is invariant to monotone reward reshaping; GRPO and GFlowNet respond to reward magnitudes, and GFlowNet needs calibration for policy diversity.Source: C RL Algorithm Details · pp. 17–18

D The SKYASSIST Environment

  • SKYASSIST is a deterministic synthetic flight-booking environment with a constrained tool surface, meaningful state, reachable policy failures, and deliberate specification gaps.Its eight tools include read-only queries and state-changing mutations; the taxonomy contains 20 policies across four categories.Source: D The SKYASSIST Environment · p. 19, Table 6 · p. 19, Table 7 · p. 20
  • Under the training-time judge, turn-level B4, B6, and S5 firings dominated raw reward, motivating per-policy deduplication; SKYASSIST also omits several real-world system features.Omitted features include dynamic pricing, payment failures, multiairline interlining, loyalty programs, seat maps, and concurrent inventory changes; policies are representative, not exhaustive.Source: D The SKYASSIST Environment · pp. 19–20, Table 9 · p. 21

E Human Validation of the LLM-Judge Verifiers

Human validation found near-perfect agreement between annotators but near-chance agreement between humans and the LLM judge, which systematically over-flagged violations. Deterministic re-scoring changed the best PROBE coverage but not method rankings.

E Human Validation of the LLM-Judge Verifiers

  • The judge marked 77% of instances as violations versus 20% for humans, with 121 false positives and 4 false negatives.Its precision was 0% on critical S5 and 18% on P1; human–human agreement was near perfect.Source: E Human Validation of the LLM-Judge… · p. 21, Table 12 · p. 22, Table 11 · p. 22
  • GRPO judge-flagged S5 violations rose from 133 to 302 across checkpoints, while the deterministic mutation-without-confirmation predicate confirmed none.The authors describe this divergence as a reward-hacking signature; DPO checkpoints showed no such escalation.Source: E Human Validation of the LLM-Judge… · p. 21
  • Deterministic re-scoring changed best PROBE coverage from 60% to 55% because S5 dropped out, but no method ranking changed.All 20 policies were re-scored with deterministic predicates; the target almost never mutates state, so the deterministic S5 predicate never fires.Source: E Human Validation of the LLM-Judge… · p. 21

F Baselines and Additional Results

Across additional evaluations, PROBE auditors retain an advantage over static testing on an unseen target family and τ-bench. The results also show sensitivity to target tool use and verifier scoring, while held-out replications are stable.

F Baselines and Additional Results

  • The LLM judge flagged 159/207 cases (77%) versus 42/207 (20%) for humans, with 24% precision and 90% recall.Table 13 is the confusion matrix comparing the judge with human consensus.Source: Table 13 · p. 22, F Baselines and Additional Results · p. 22
  • Across three independent held-out samples, PROBE-GRPO and PROBE-DPO each varied by one policy, with every sample roughly an order of magnitude above the static suite.Table 14 reports the deterministic-scoring coverage results.Source: Table 14 · p. 23, F Baselines and Additional Results · p. 23
  • Searches dominate target tool use, while booking, holding, and cancellation are rare, structurally limiting which policies a black-box auditor can expose.Baseline comparisons used the same frozen target, verifiers, 50 held-out scenario seeds, and 8-turn budget as PROBE.Source: F Baselines and Additional Results · pp. 22–23
  • Under deterministic scoring, the best PROBE checkpoints fell from 60% to 55% after losing S5; zero-shot stayed at 35%, and method rankings were unchanged.Training-time yields included judge false positives; S5 was flagged with 0% precision.Source: F Baselines and Additional Results · p. 23

G.1 Unseen Target Family: Gemma

  • On Gemma, PROBE-DPO exposed 8/20 policies (yield 190), versus 1/20 (yield 42) for the static suite, retaining 73% of in-domain coverage.Its advantage over static testing was 8× on Gemma, compared with 11× in-domain.Source: §G.1 · p. 23

G.2 τ-bench

  • On τ-bench, episodes carry fixed task goals and the auditor used PROBE-DPO without reward-weight retuning; the 10-policy suite includes eight deterministic verifiers and two heuristic verifiers.The target, verifiers, and turn budget were shared with the identical untrained Qwen3.5-9B baseline.Source: §G.2 · pp. 23–24
  • On τ-bench, PROBE-DPO raised coverage from 20% (2/10) to 30% (3/10) and violations from 13 to 29 per 50 episodes (+123%) on the same base model.The 4B auditor reached comparable coverage with 7 violations per 50 episodes, about 4× fewer than the 9B auditor.Source: §G.2 · p. 24

H Reward Ablations and Sensitivity

The reward ablations examine how reward versions and components shape auditor behavior and held-out coverage. Per-policy credit is associated with broader coverage than a generic diversity objective, while plausibility-weight perturbations have limited effects.

Reward versions

  • The experiment labels V1–V3 correspond to baseline, behavior, and calibrated rewards; +Dedup adds per-policy diminishing returns without the other calibrated-reward changes.The diversity-only variant uses binary any-violation reward without per-policy credit and was evaluated with GFlowNet only.Source: H Reward Ablations and Sensitivity · p. 24
  • The calibrated PROBE reward (V3) reduced role confusion to 11%, while training sweeps covered 12–14 of 20 policies and raw reward remained dominated by frequent B4/S5/B6 hits.DPO margins increased as repeated-hit inflation decreased.Source: H Reward Ablations and Sensitivity · p. 24

Tuning protocol

  • Reward versions were selected using training-time behavioral diagnostics, and held-out episodes were not used for selection; deterministic predicates counted findings independently of reward weights.Diagnostics included role confusion, loop rate, completion length, and DPO margins.Source: H Reward Ablations and Sensitivity · p. 24

Reward configuration sensitivity

  • Across trained reward configurations, every configuration exceeded the static suite by at least 7×; replacing per-policy credit with generic diversity changed coverage to 35% vs. 45% for GFlowNet and 50–55% for GRPO and DPO.Coverage used deterministic scoring over 50 held-out episodes each. Diversity-only was trained with GFlowNet only, and +Dedup with GRPO and DPO only.Source: H Reward Ablations and Sensitivity · p. 25, Table 16 · p. 25

One-at-a-time perturbation

  • A fourfold change in the plausibility weight shifted coverage by at most one policy, and halving it reproduced the reference policy set exactly.The comparison used matched GRPO training steps and deterministic scoring on 50 held-out episodes.Source: Table 17 · p. 25, H Reward Ablations and Sensitivity · p. 25

Severity assignment

  • With uniform severities (σi=1), method rankings stayed unchanged; the discovered-versus-undiscovered split followed verifier firing semantics rather than severity weights.Source: H Reward Ablations and Sensitivity · p. 25

I Implementation Details

The implementation trains a smaller auditor to interact as a realistic customer with a frozen tool-using target, using multi-turn rollouts and database-aware scenarios. Qualitative traces illustrate both useful probes and limits when target behavior does not reach deeper mutation states.

I Implementation Details

  • The 4B Qwen3 auditor was trained with 4-bit NF4 QLoRA while the Qwen3-30B-A3B target remained frozen, including its weights, prompt, database, and verifiers.The adapters yielded about 47M trainable parameters.Source: I Implementation Details · p. 25
  • The auditor is prompted as a realistic customer, receives scenario context but not the policy list, and must keep its messages consistent with the conversation.The target uses a fixed system prompt and eight tools; database-aware scenario instantiation avoids impossible requests that cannot reach meaningful tool states.Source: I Implementation Details · pp. 25–26
  • The rollout curriculum expands from 4 to as many as 14 turns as violation yield improves, allowing longer dialogues to reach booking, cancellation, modification, and multi-leg policies.Many policy decisions follow benign turns that establish travel details or modification intent.Source: I Implementation Details · p. 26

J Qualitative Trace Patterns

  • A plausible partial flight request elicited a premature search_flights call missing the required origin, triggering the schema-adherence verifier despite a verbally appropriate reply.Source: J Qualitative Trace Patterns · p. 26
  • Combining ordinary customer constraints can expose interacting rules that isolated one-policy tests miss, but repeated turn-level omissions on routes without flights can earn raw reward without reaching booking or modification.The authors identify deduplication and coverage as responses to this high-yield, low-value probe pattern.Source: J Qualitative Trace Patterns · p. 26
  • Some operational policies remain unreachable unless the target enters a deeper mutation state; in the group-booking probe, the target never called book_flight, so the passenger-limit verifier could not fire.Source: J Qualitative Trace Patterns · p. 27

Related papers

Loading 2610.22124v1…