PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaGuard: A Dynamic Guardian Model With User-Defined Policies

Monte Hoover, Vatsal Baherwani, Neel Jain, Khalid Saifullah, Joseph Vincent, Chirag Jain, Melissa Kazemi Rad, C. Bayan Bruss, Ashwinee Panda, Tom Goldstein

arXiv:2509.02563v3cs.LGcs.CL

TL;DR

Guardian models based on static harm categories may not reflect application-specific criteria for undesirable behavior. DynaGuard uses user-defined policies, with DynaBench providing training and evaluation data; the models perform strongly across safety and flexible-policy evaluations, including in fast non-CoT mode.

  • Problem

    Static harm categories do not capture application-dependent criteria for undesirable behavior, such as business-specific or context-sensitive cases.

    Source: §1 · pp. 1–2

  • Method

    DynaGuard models assess compliance with user-defined policies, trained using DynaBench’s bespoke policies and conversations, with optional reasoning traces before classification.

    Source: §1 · p. 2, §3.1 · p. 4, §3.4 · p. 5

  • Results

    DynaGuard achieves strong performance across traditional safety benchmarks and unseen DynaBench rules; without CoT, DynaGuard-8B outperforms GPT-4o-mini and DynaGuard-1.7B outperforms existing guardian models.

    Source: §4 · pp. 7–8

  • Takeaways & Limitations

    DynaGuard offers a flexible guardian model that performs well on guardian tasks despite its small size and latency.

    Source: §5 · p. 9

    Further work is needed to determine how explanations integrate into multi-agent recovery and affect human trust and usability in interactive or assistive settings.

    Source: §5 · p. 10

Abstract

from arXiv · show

Guardian models play a crucial role in ensuring the safety and ethical behavior of user-facing AI applications by enforcing guardrails and detecting harmful content. While standard guardian models are limited to predefined, static harm categories, we introduce DynaGuard, a suite of dynamic guardian models offering novel flexibility by evaluating text based on user-defined policies, and DynaBench, a dataset for training and evaluating dynamic guardian models. Our models provide both rapid detection of policy violations and a chain-of-thought reasoning option that articulate and justify model outputs. Critically, DynaGuard not only surpasses static models in detection accuracy on traditional safety categories, but is competitive with frontier reasoning models on free-form policy violations, all in a fraction of the time. This makes DynaGuard an critical tool for language model guardrails.

1 Introduction

Existing guardian models rely on static harm categories, although what counts as harmful can depend on the application. DynaGuard supports user-defined policies, and DynaBench supplies data for training and evaluating such guardians.

1 Introduction

  • Static harm categories can miss application-specific risks: a chatbot’s mistaken refund offer led to legal responsibility, while medical and retrieval settings require context-sensitive distinctions.For example, medical guardrails may block sexual content without blocking anatomy discussions, and retrieval-augmented models should distinguish planning violence from discussing retrieved news.Source: §1 · pp. 1–2
  • DynaGuard replaces fixed harm categories with user-defined policies and provides pass/fail judgments plus natural-language explanations that can support agent correction.The framework also identifies fast inference and open weights as desired properties for adoption across industrial settings.Source: Figure 1 · p. 1, §1 · p. 2
  • DynaBench contains 40K bespoke policies with simulated conversations showing adherence and violations; its evaluation set includes domain-specific, human-written policies beyond the training scope.LlamaGuard3 achieves 13.1% F1 on the test set, which includes complex multi-turn violations and adversarial jailbreaks.Source: §1 · p. 2
  • Training on DynaBench improves guardian capability, and the open-source 8B DynaGuard model outperforms GPT-4o-mini on the DynaBench evaluation set with lower cost and latency.The introduction also frames dynamic policies, interpretability, fast inference, and open weights as desired guardian-model traits.Source: §1 · p. 2, Table 1 · p. 2

2 Related Work

Prior guardian models span fixed safety taxonomies, custom rule configurations, reasoning-based approaches, and smaller models, while related datasets provide varied harm labels and conversation settings. This work extends dataset efforts toward turn-level compliance across diverse real-world policies and rules.

2.1 Guardian Models

  • LlamaGuard uses a fixed safety taxonomy and has limited zero-shot generalization beyond toxicity-related domains, while later work explores custom rules, reasoning, and smaller guardians.Custom safety configurations remain limited to the safety domain; other work addresses reliability and multimodal moderation.Source: §2.1 · pp. 2–3

2.2 Compliance-related datasets

  • Compliance datasets range from labeled harm and helpfulness examples to single-turn real-world or synthetic dialogues with adversarial, user-input, and response-harm labels.Aegis2.0 adds commercial-use categories such as copyright, trademark, high-risk government decisions, and unauthorized advice.Source: §2.2 · p. 3
  • DynaBench’s related contribution is turn-level compliance evaluation across diverse real-world policies and rules.Source: §2.2 · p. 4

3 Creating the DynaBench Dataset

DynaBench pairs multi-turn policy-compliance dialogues with specialized, domain-specific rules, using a hybrid construction pipeline and a separate handcrafted test set.

3 Creating the DynaBench Dataset

  • The hybrid pipeline yields a 61.5k-sample training set and a separate handcrafted 543-example test set of labeled multi-turn dialogues for specialized policies.The dataset targets policies beyond traditional safety domains such as toxicity or bias.Source: §3 · p. 4

3.1 Constructing the Rule Bank and Policies

  • About 500 hand-written rules expand to 5,000 curated rules and 40,000 policies; test cases pair 12 business-impact categories with 16 failure modes and disjoint train/test rules.Policy sizes follow an exponential distribution, with a median of three rules and a maximum of 86; rules are combined thematically and paraphrased.Source: §3.1 · p. 4

3.2 Dialogue Generation

  • LLMs generate multi-turn, profile-diverse dialogues; compliance labels are produced rule-by-rule, and four safety datasets are converted to policy format.GPT-4o labels each rule separately, while smaller models generate dialogues and larger models provide labels and reasoning traces.Source: §3.2 · pp. 4–5

3.3 Validating DynaBench Training Set

  • Three annotators’ manual review of a challenging 743-example training subset yields Cohen’s Kappa 0.85, versus 0.72 for WildGuard’s response-refusal task.The validation subset has median policy size 10 and six dialogue turns, compared with four and two in the original training set.Source: §3.3 · p. 5

3.4 Model Training

  • Qwen3 guardian models train on a mixture of 40,000 DynaBench and 40,000 safety-dataset samples, followed by one epoch of SFT and GRPO on 11,000 samples.One-third of examples use reasoning traces before the compliance label; the remaining two-thirds place an answer before an actionable explanation.Source: §3.4 · pp. 5–6

3.5 Mathematical formulation for the different training schemes

  • C-SFT maps rules and dialogue to a compliance label; CT-SFT adds thinking traces for one-third of samples, while GRPO generates traces and labels for evaluator-scored optimization.GRPO’s scalar evaluator reward is assigned to generated trace-label pairs; its advantage subtracts expected reward.Source: §3.5 · p. 6

3.6 Evaluation

  • Evaluation toggles reasoning or fast inference with <think>/<answer> tags, compares baselines under documented prompts, and repeats datasets with up to six seeds.Base Qwen and API models were prompted for reasoning; guardian models used documented system prompts.Source: §3.6 · p. 6
  • DynaGuard’s non-CoT mode is competitive with reasoning mode, differing by only 1.3% in F1, and <explanation> elicits an actionable explanation.The two modes provide fast inference and reasoning options.Source: §3.6 · p. 6

4 Results

DynaGuard’s SFT + GRPO recipe on mixed safety and DynaBench data leads across the evaluated benchmarks, while non-CoT models remain competitive. Ablations, unseen-policy tests, and model-family results further examine the recipe and its flexibility.

4 Results

  • DynaGuard-8B has the best average across all tasks, and SFT + GRPO on a 50/50 safety–DynaBench mix yields the best overall performance across benchmarks.Without CoT, DynaGuard-8B outperforms GPT-4o-mini, while DynaGuard-1.7B outperforms all existing guardian models.Source: Table 3 · p. 7, §4 · p. 8
  • DynaBench training increases combined WildGuard and DynaBench F1, reasoning training further improves it, but DynaBench-only training does not reach state-of-the-art out-of-distribution safety detection.DynaBench training data contains no safety ontology; the authors report that its generalist training nevertheless generalizes to domains where common guardian models do not.Source: §4 · p. 8, Table 5 · p. 8
  • DynaGuard is the only evaluated model reported to handle unseen IFEval policies in the out-of-distribution setting.The setup uses IFEval instructions as policies, with explanations prompting Ministral-8B to regenerate violated responses.Source: §4 · p. 8

4.1 Effectiveness Across Model Families

  • The safety-plus-DynaBench SFT and GRPO recipe extends across multiple model families, with reported family values averaging base-to-finetuned deltas across available 1B–8B sizes.The available passage does not provide the table’s delta values.Source: §4.1 · pp. 8–9, Table 7 · p. 9
  • Pairing Ministral-8B with DynaGuard to correct detected instruction violations improves IFEval performance.The case study shows GPT-4.1-mini revising a response to comply after DynaGuard explains a detected violation.Source: Table 6 · p. 9, §4.1 · p. 9

5 Conclusion

The paper presents DynaBench for training and evaluating guardian models, and reports that DynaGuard performs well on flexible guardian tasks despite small size and latency. It also identifies open questions about explanations and adaptation to new use cases.

5 Conclusion

  • DynaBench is introduced as a challenging dataset for training and evaluating guardian models.DynaGuard is reported to achieve state-of-the-art performance on flexible guardian tasks despite its small size and latency.Source: §5 · p. 9
  • Further work is needed to understand how explanations should integrate with multi-agent recovery strategies and affect human trust and usability in interactive or assistive settings.The authors also anticipate updating the model as new use cases emerge.Source: §5 · p. 10
  • The authors hope flexible guardian capabilities will lead to broader adoption of agentic paradigms for model safety.This is stated as an expectation, not a demonstrated outcome.Source: §5 · p. 10

A.1 Constructing a Post-Hoc Taxonomy

The authors categorize 5,000 rules into five post-hoc categories that do not overlap with established harm-based taxonomies. Their seed-rule process uses miscellaneous and industry-specific categories to encourage policy diversity.

A.1 Constructing a Post-Hoc Taxonomy

  • Post-hoc categorization of 5,000 rules yields five categories outside established harm-based taxonomies: User Experience, Regulatory Compliance, Content Controls, Transactions, and Agentic Tasks.The objective is to evaluate guardian models across behavioral categories beyond predefined limitations.Source: §A.1 · p. 14
  • Examples span company phrasing and awards, technical specifications and consent, religion and warfare, discounts and returns, and survey links or sales referrals.These examples correspond to the five category passages supplied.Source: §A.1 · p. 14
  • Writers of the 500 initial seed rules receive 25 miscellaneous categories and one industry-specific category with 204 options to encourage diversity from the outset.The passage introduces the 25 miscellaneous categories but does not enumerate them here.Source: §A.1 · p. 14

A.2 Dataset Label Validation during Data Development

The data-development pipeline iteratively checks synthetic labels against human judgments and refines generation before producing the full dataset. Subsequent review uses one annotation per sampled training example and multiple annotators for sampled test examples.

A.2 Dataset Label Validation during Data Development

  • Four development iterations each validate 40 samples, filtering ambiguous policies and optimizing the labeling prompt until human-label agreement reaches 90%.Source: §A.2 · p. 14
  • Three annotators review 743 training samples and 25 test samples; training examples receive one human annotation each, while test examples receive multiple annotations for inter-rater agreement.Source: §A.2 · pp. 14–15
  • The authors report high test-set inter-rater agreement and say training-set gains on that high-agreement test set support the training data’s efficacy.Source: §A.2 · p. 15

A.3 DynaBench Dataset Characteristics

DynaBench samples are organized around business impacts and failure modes, with each sample assigned one of each. The dataset spans varied rule counts and dialogue lengths across train and test.

A.3 DynaBench Dataset Characteristics

  • Each sample is annotated with one business impact and one intended failure mode, drawn from 12 impact categories and 16 failure modes.The categories guide sample writing.Source: §A.3 · p. 15, Figure 4 · p. 16, Figure 5 · p. 16
  • Failure modes include policies that exceed or relax traditional safety categories, precise counting, multi-turn reasoning hops, and combinations of challenges.The examples are a subset of the failure-mode categories.Source: §A.3 · p. 15
  • Training policies contain up to 86 rules and dialogues up to 27 turns, while test policies reach 91 rules and dialogues up to 13 turns.Figure 6 shows the distributions for train (top row) and test (bottom row).Source: Figure 6 · p. 17, §A.3 · p. 17

A.4 Error analysis

The error analysis examines accuracy across subsets grouped by failure mode and explores how model capabilities vary with model size. Its tables identify failure modes with extreme error rates and lengths beyond which accuracy falls below 50%.

A.4 Error analysis

  • The analysis uses accuracy on failure-mode subsets to examine difficult areas and capability changes with model size.Tables 10 and 11 provide examples of these error patterns.Source: §A.4 · p. 17
  • Table 10 lists the three highest and three lowest error-rate failure modes for DynaGuard-8B.The supplied table caption does not state the categories or their error-rate values.Source: Table 10 · p. 17
  • Table 11 identifies the longest conversations and policies each model handles before accuracy drops below 50%.The supplied caption does not give the corresponding lengths or model-specific results.Source: Table 11 · p. 17

A.5 Synthetic Training Example

A synthetic training example pairs a travel-coverage rule with a multi-turn conversation and a reasoning trace that judges whether the agent followed the rule.

A.5 Synthetic Training Example

  • The example's rule requires immediate notice that standard plans typically exclude medical care outside the United States, subject to specified emergency exceptions.The rule applies when a user mentions international travel.Source: §A.5 · p. 18
  • The user describes a backpacking trip through Europe and asks about travel insurance, while the agent discusses providers without explaining international medical coverage.The dialogue supplies the scenario evaluated against the rule.Source: §A.5 · p. 18
  • The reasoning trace marks the rule violated because the agent does not provide the required coverage information.It identifies the user's trip as international travel and contrasts the rule with the agent's responses.Source: §A.5 · p. 18

A.6 Test Set Examples

The handcrafted test examples are nuanced scenarios set in real-world contexts, and a sampled portion received 100% human inter-rater agreement. They illustrate policy-sensitive dialogues involving member appointment information and prohibited tax-haven topics.

A.6 Test Set Examples

  • The test set uses human-written scenarios intended to be nuanced and relevant to real-world settings, with 100% inter-rater agreement on a sampled portion.The examples that follow illustrate the set's scenarios.Source: §A.6 · p. 18
  • One example pairs a prohibition on revealing gym members' appointment times with dialogue asking for a member's schedule and conditional appointment options.The dialogue includes requests conditioned on whether the member has a session on particular dates.Source: §A.6 · pp. 18–19
  • Another example pairs a ban on tax-haven topics such as shell companies with a conversation that introduces a Cayman Islands company selling seashells.The conversation explicitly calls the company “Cayman Shells Ltd.”Source: §A.6 · p. 19

A.7 System Prompts

This appendix section documents prompts for generating and labeling policy-conditioned dialogues and for assessing policy violations, while also presenting supplementary findings on IFEval correction, training choices, evaluation reproducibility, failure modes, and societal benefits and risks.

A.7.1 Data Generation Prompts

  • Dialogue generation prompts create realistic business conversations from user and agent descriptions, with a supplied rule and a target of compliance or violation.The prompts provide the rule and descriptions as inputs to the transcript task.Source: §A.7.1 · pp. 19–20
  • Transcripts use user-agent turns and specified round counts; internal task records appear only when needed and should remain unknown to the user.Records can show actions such as API calls or notifications to other parties.Source: §A.7.1 · p. 20
  • Generation scenarios vary whether users behave normally or try to trick the agent, and whether the agent resists, falls for the trick, or violates part of a rule.Some violation examples are requested to partly follow, or nearly follow, the rule before breaking it.Source: §A.7.1 · p. 20
  • The labeling prompt requests JSON with discussion, explanation, violation label, and ambiguity score, using rule or transcript quotations where possible.It defines labels as VIOLATED or NOT VIOLATED and ambiguity from 0 (unambiguous) to 10 (fully ambiguous).Source: §A.7.1 · pp. 20–21
  • One labeling example treats a detailed employee inspection process as an internal policy, marking its disclosure VIOLATED with ambiguity 4.The example notes that the process is not explicitly called a policy, making the inference ambiguous.Source: §A.7.1 · p. 24

A.7.2 Guardian Model Prompts

  • Guardian prompts assess messages against safety categories and a supplied policy, with outputs ranging from safe/unsafe ratings to JSON listing applicable categories.The prompts specify categories and distinguish agent-only assessment from a prompt that also rates user messages.Source: §A.7.2 · pp. 25–26
  • GuardReasoner separates policy assessment into whether the request applies, whether the assistant refuses or complies, and whether its response violates the policy.Source: §A.7.2 · pp. 26–27
  • The guardian prompts define policy-conditioned classification tasks, including checking request applicability, refusal status, and whether the assistant response violates policy.The prompts require step-by-step reasoning and, in one format, an answer beginning with “Yes” or “No.”Source: §A.7.2 · pp. 26–27

A.8 IFEval Analysis

  • 32 of 232 initial IFEval failures were corrected (13.8% improvement rate) after DynaGuard-8B identified 157 failures.The initial failures came from Ministral-8B on 541 samples; sampled explanations were human-coherent, with the greatest improvement for bulleted lists (18.8%) and none for JSON formatting (0%).Source: Table 12 · p. 27, §A.8 · p. 27

A.9 Training Details

  • At 11k GRPO samples, validation performance plateaued, so this checkpoint was selected for the final DynaGuard models.Beyond 11k, GRPO improved performance on the DynaBench validation subset while decreasing it on the Safety subset.Source: §A.9 · p. 29

A.10 Evaluation Details

  • NemoGuard evaluation reproduced the reported scores except for WildGuard F1, which was 64.5 rather than the reported 77.5.The authors found JSON responses with a “User Safety” entry but no “Response Safety” entry.Source: §A.10 · p. 29

A.11 Additional Failure Mode Analysis

  • Across failure-mode subsets, some models showed decreased accuracy as the annotated failure mode became more difficult.Figures 7–16 report evaluation results broken out by benchmark subsets annotated with specific failure modes.Source: §A.11 · p. 29

A.12 Societal Impacts

  • DynaBench and DynaGuard are intended to let practitioners tailor LLM safety guardrails, but their imperfect accuracy requires care in deployment.The paper highlights potential use with populations such as young students or people recovering from trauma.Source: §A.12 · p. 29
Loading 2509.02563v3…