PaperlayerView exact revision on arXiv

Source-linked AI summary

PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails

Mingyang Song, Luxin Xu, Haoyu Sun, Minzhou Pan, Yu Cheng, Bo Li

arXiv:2607.05910v1cs.CVcs.AIcs.CL

TL;DR

Existing image guardrails and benchmarks often assume fixed safety labels, but deployments require decisions that adapt to the active policy. This paper introduces PolicyShiftBench and PolicyShiftGuard with boundary-paired policy supervision, achieving policy-sensitive gains and strong transfer while improving efficiency.

  • Problem

    Existing benchmarks and methods do not fully evaluate or optimize image guardrails’ ability to adapt decisions to changing runtime policies.

  • Method

    The paper introduces PolicyShiftBench and trains PolicyShiftGuard with randomized policy supervision and boundary-paired comparisons that separate matched blocking and passing policies.

  • Results

    PolicyShiftGuard transfers well to external safety benchmarks and improves the latency–performance trade-off, reaching 76.9 Avg. F1 on PolicyShiftBench.

  • Takeaways & Limitations

    Multimodal guardrails should be evaluated by whether decisions follow the enforced policy, rather than fixed safe/unsafe recognition alone.

  • Takeaways & Limitations

    The work evaluates static images paired with explicit policy text, not video, audio, multi-turn conversations, or dynamic web-page contexts.

Abstract

from arXiv · show

Image guardrails are typically trained and evaluated under a fixed safety policy, implicitly treating safety as an intrinsic property of an image. Real deployments are different: the same image may be allowed in one product, restricted in another, and newly disallowed when a policy boundary changes. We study policy-adaptive image guardrailing, where a model must decide whether an image violates the currently supplied policy and generalize to held-out policy definitions. We introduce PolicyShiftBench, a comprehensive benchmark with 2,000 policy-discriminative instances over 265 images, where each image is paired with 7.55 policy-conditioned prompts on average to test whether models adapt to the active policy rather than relying on image-level safety priors. We then propose PolicyShiftGuard, a compact policy-conditioned guardrail trained with a two-stage training recipe that combines Randomized Policy SFT (RP-SFT) with Boundary-Pair Policy Adaptation (BP-Adapt). BP-Adapt trains matched prompts for the same image and risk category using standard label supervision and a pairwise comparison loss that separates blocking policies from passing policies. Experiments show that existing VLMs and specialized guardrails remain brittle under policy shifts, while PolicyShiftGuard substantially improves policy-sensitive performance. The 7B model achieves SOTA performance of 76.9 Avg. F1 and 72.1 Avg. PSS on PolicyShiftBench, transfers well to UnSafeBench and SafeEditBench, and improves the latency-performance trade-off with a concise output format. Ablations confirm that matched pass/block boundary pairs are essential for stable policy adaptation.

1 Introduction

Policy-adaptive image guardrailing requires models to follow changing runtime policies rather than treat safety as an intrinsic image property, yet existing benchmarks and methods inadequately test this capability. The paper introduces PolicyShiftBench and PolicyShiftGuard to evaluate and improve policy-sensitive decisions through matched policy conditions and two-stage training.

  • Motivation and gap: Existing fixed-taxonomy benchmarks make it difficult to distinguish generic unsafe-content recognition from following the active policy.Recent policy-conditioned evaluations also remain insufficiently fine-grained for measuring policy shifts.
  • PolicyShiftBench: PolicyShiftBench covers seven risk categories, five moderation scenarios, 28 policy variants, and 2,000 instances over 265 images.Each image has 7.55 policy-conditioned prompts on average, and 262 of 265 images appear with both pass and block labels.
  • PolicyShiftGuard: PolicyShiftGuard combines auditable policy-rule-based data curation with Randomized Policy SFT and Boundary-Pair Policy Adaptation.RP-SFT teaches full policy-bundle following with concise structured decisions, while BP-Adapt explicitly optimizes handling policy shifts.
  • Results: Experiments show that current VLMs and specialized guardrails remain brittle under policy shifts, with ordinary F1 not necessarily indicating policy-shift sensitivity.Scaling model size helps but does not resolve policy adaptation, whereas PolicyShiftGuard substantially closes the performance gap.
  • Results: PolicyShiftGuard also transfers strongly to external safety benchmarks and improves the latency–performance trade-off for practical deployment.The framework is presented as an end-to-end approach combining benchmark-driven evaluation with policy-adaptive guardrail training.

2 PolicyShiftBench: A Benchmark for Policy-Adaptive Image Guardrailing

PolicyShiftBench evaluates whether image guardrails adapt to deployment-time policy changes rather than treating safety as intrinsic to an image. It combines scenario-grounded policies, auditable attribute-based labels, and policy-discriminative boundary examples across a multi-split benchmark.

  • Benchmark design: The benchmark spans seven risk categories, each paired with realistic moderation scenarios that define diverse policy variants.A policy is the intersection of one risk category and one scenario, rather than an arbitrary strictness level.
  • Policy-conditioned labeling: The same image attribute vector can receive different acceptability labels because the active policy determines which content passes or is blocked.For example, medical nudity may pass in medical or educational contexts but fail under a strict family-safe policy.
  • Benchmark scale: 2,000 policy-discriminative instances cover 265 unique images and 28 evaluation policy variants, split evenly between Adaptive and held-out-policy Shift evaluations.The Adaptive Split contains 1,000 instances over 130 images and 16 policies; the Shift Split contains 1,000 instances over 152 images and 12 held-out policies.
  • Data construction: Images are converted into structured metadata through majority-voted attribute annotations, after which executable policy rules deterministically produce category decisions and final labels.Each prompt activates one policy per category, and the final label is unsafe if any category blocks the image; otherwise it is safe.
  • Policy-discriminative examples: Boundary instances hold the image and target category fixed while changing the active policy, forcing models to read the policy because the visual evidence is identical.These examples prevent solving by recognizing generic unsafe cues and instead test whether attributes cross the current policy boundary.
  • Quality control: 97.50% of 475,569 canonical field decisions are unanimous across three annotators, while 2.50% are resolved by majority vote.Quality control also uses deterministic validators, human auditing, image-disjoint partitions, mini test sets, and visual review packages.

3 PolicyShiftGuard Training Method

PolicyShiftGuard uses a two-stage training recipe: RP-SFT teaches policy-conditioned task following and structured outputs, while BP-Adapt trains matched pass/block pairs to make decisions change with the active policy. The method combines randomized policy presentation with boundary-aware objectives that directly separate blocking and passing policies under identical visual evidence.

  • Randomized Policy SFT (RP-SFT): RP-SFT trains on disjoint images with runtime policy bundles, teaching the model to follow the evaluation policy schema and produce deterministic structured guardrail decisions.Each example pairs an image with a seven-policy bundle and a target computed by executing the active rules.
  • Randomized Policy SFT (RP-SFT): Policy randomization changes policy order, surface identifiers, and category-slot assignments, preventing shortcuts based on fixed positions or textual templates.The same transformation is applied to the target category identifier, requiring the model to bind visual evidence to the active rule.
  • Boundary-Pair Policy Adaptation (BP-Adapt): BP-Adapt uses boundary pairs that hold the image and visual evidence fixed while changing the active policy and correct label.The post-training source is independent of both Adaptive and Shift evaluation branches.
  • Boundary-Pair Policy Adaptation (BP-Adapt): Boundary-aware margins complement supervised objectives by separating answer strings, safe/unsafe prefixes, violated categories, and matched policy decisions.The pair loss forces the blocking policy to receive a higher unsafe score than the passing policy for identical visual evidence.
  • Boundary-Pair Policy Adaptation (BP-Adapt): BP-Adapt preserves pass/block counterfactuals rather than merely continuing SFT, directly optimizing the policy-conditioned decision margin used in policy-shift evaluation.This targets failures where guardrails block visible risk cues regardless of policy or pass images under coarse global rules.

4 Experiments

Experiments evaluate policy following and held-out-policy generalization across POLICYSHIFTBENCH, with cross-benchmark tests on UnSafeBench and SafeEditBench. Results show that policy adaptation remains distinct from visual safety recognition, while PolicyShiftGuard improves policy-sensitive performance and latency.

  • Benchmarks and metrics: POLICYSHIFTBENCH evaluates policy following on the Adaptive Split and held-out policy generalization on the Shift Split, with transfer tests on UnSafeBench and SafeEditBench.Metrics include Accuracy, F1, and PSS; PSS measures correct decisions on same-image policy flips.
  • Policy-adaptive performance: 76.9 Avg. F1 and 72.1 Avg. PSS are achieved by PolicyShiftGuard-7B, improving Qwen2.5-VL-7B from 20.6 and 4.8, respectively.The 7B model achieves the best overall policy-adaptive performance among evaluated systems.
  • Latency-performance tradeoff: 163.9 ms versus 273.3 ms reduces PolicyShiftGuard-7B latency relative to base Qwen2.5-VL-7B while preserving accuracy.Its concise output typically resolves decisions within five tokens, with the first token determining binary-only judgments.
  • Findings: 59.2 Avg. F1 versus 3.2 Avg. PSS for GuardReasoner-VL-3B shows that visual risk recognition does not ensure policy-shift sensitivity.SafeGuard-VL-RL-7B similarly reaches 51.0 Avg. F1 but only 4.0 Avg. PSS.
  • Findings: 49.4 Avg. F1 at 72B versus 20.6 Avg. F1 at 7B shows scaling helps Qwen2.5-VL, but its Avg. PSS reaches only 27.4.Qwen3.5-35B-A3B likewise has only 18.6 Avg. PSS, indicating that scale alone does not solve adaptation.
  • Findings: Risk categories differ in difficulty: nudity and violence are easier, whereas regulated goods, IP and brand safety, privacy and PII, and text-in-image safety require finer attribute extraction and policy exceptions.The results motivate labels based on explicit image attributes and executable policy rules rather than a single fixed safety taxonomy.

5 Ablation Study

Ablations show that policy-adaptive guardrailing depends on concise decision optimization and explicit matched boundary-pair supervision, not merely randomized policies or additional data. Boundary-pair training remains effective across nearby loss-weight settings.

  • Randomized policy SFT: RP-SFT improves stage-1 Avg. F1 over standard SFT by 1.2 points for 3B and 7.2 points for 7B, but randomization alone does not consistently improve policy adaptation.For 7B, Shift PSS decreases by 9.7 points versus standard SFT.
  • Thinking and RL: No-think training outperforms thinking-mode training in both stages, with Avg. F1 gains of +4.3 for 3B and +8.9 for 7B in stage 1.The advantage increases to +5.6 for 3B and +12.3 for 7B in stage 2, while thinking mode produces longer outputs and higher inference cost.
  • Boundary-pair objective: Removing Lpair substantially reduces Avg. F1 and Avg. PSS for both 3B and 7B, while direct continuation on additional boundary prompts underperforms corresponding stage-1 baselines.The results indicate that gains arise from explicitly separating matched pass/block decisions for the same image under different policies, rather than from more post-training data alone.
  • Loss-weight sensitivity: Across tested λw, λp, and λl ranges, Avg. F1 and Avg. PSS fluctuate only mildly, indicating that BP-Adapt is not overly sensitive to a narrow loss-weight configuration.The loss-weight ablation varies λw, λp, and λl on Qwen2.5-VL-7B.

6 Related Work

Prior visual safety benchmarks generally evaluate under fixed policies, although newer guardrails explore explicit policy conditioning. Related work also studies adaptive safety policies in text moderation and whether explicit reasoning improves guardrail decisions.

  • Visual safety benchmarks: Traditional visual safety benchmarks use fixed taxonomies or static harmfulness definitions, exposing failures under distribution shift, calibration error, and jailbreak-style attacks.UnsafeBench, MM-SafetyBench, VSCBench, and related work generally keep the governing policy fixed during evaluation.
  • Policy-conditioned guardrails: Recent visual guardrails introduce explicit policy conditioning, including LlavaGuard’s support for customizable visual safety taxonomies.
  • Policy-conditioned benchmarks: BENCH evaluates category-compositional runtime policy bundles using executable rules, held-out policies, and unsafe predictions that name violated categories.Its labels derive from fine-grained visual attributes and distinguish in-domain from held-out policies.
  • Adaptive safety policies: Text moderation research examines adaptation to community rules, user-defined policies, unseen taxonomies, and domain-specific standards.
  • Reasoning-based guardrails: Another research line investigates whether explicit reasoning improves guardrails, including ThinkGuard, Safety Through Reasoning, GuardReasoner-VL, and MSR-Align.

7 Conclusion · Appendix

The paper introduces PolicyShiftBench for evaluating image guardrails under changing runtime policies and PolicyShiftGuard, a compact guardrail trained with boundary-paired supervision. Results show improved policy-sensitive performance with efficient decoding, motivating policy-grounded multimodal guardrail evaluation.

  • 7 Conclusion: PolicyShiftBench evaluates image guardrails when the same image requires different decisions under different runtime policies.It targets policy-conditioned decisions rather than a fixed image-level safety judgment.
  • 7 Conclusion: PolicyShiftGuard is a compact image guardrail trained with boundary-paired supervision.The training approach is designed around matched policy boundaries.
  • 7 Conclusion: Existing models often recognize risky content without reliably following shifted policies.This exposes a gap between risk recognition and policy adherence.
  • 7 Conclusion: PolicyShiftGuard improves policy-sensitive performance under changing runtime policies.The conclusion attributes this improvement to the proposed guardrail.
  • 7 Conclusion: PolicyShiftGuard preserves efficient decoding while improving policy-sensitive guardrailing.The result combines stronger policy sensitivity with decoding efficiency.
  • 7 Conclusion: Multimodal guardrails should be evaluated by whether decisions are grounded in the enforced policy, not only fixed safe/unsafe recognition.This evaluation principle follows from the observed brittleness under policy shifts.

A Limitations … C Policy and Rule Catalog

The paper defines an auditable, policy-adaptive benchmark by separating image attributes from executable policy rules and sampling policy-sensitive boundary cases. It also documents dataset splits, category and policy coverage, annotation aggregation, and limitations on modality, language, and policy scope.

  • A Limitations: The study is limited to static images paired with explicit English-style policies, excluding video, audio, multi-turn dialogue, dynamic webpages, and broader policy formats.The authors identify temporal reasoning, dialogue-state tracking, layout understanding, multilingual coverage, legal-document-style policies, and long user-defined policies as future extensions.
  • A Limitations: PolicyShiftBench covers seven risk categories and 28 evaluation policy variants, but its finite catalog does not yet span more industries, jurisdictions, platform norms, or regional moderation scenarios.The benchmark includes compositional runtime policy bundles and a held-out Shift split.
  • B.1 Construction pipeline: PolicyShiftBench separates visual perception from policy judgment by annotating observable triggers and exemptions before applying deterministic category rules and composing runtime policy bundles.Triggers include exposed genitalia, hard drugs, license plates, and scam text; exemptions include medical education, art, news, laboratory, and document-scan contexts.
  • B.2 Split summary: The data uses Eval-Adaptive, Eval-Shift, and BP-Adapt Augmentation branches, with 9,816 released training instances across RP-SFT, aligned think diagnostics, and boundary-pair examples.Eval-Shift holds out policy definitions, while BP-Adapt augmentation policies are training-only; training and evaluation images are disjoint, and the two evaluation tracks share 17 images.
  • B.3 Category coverage: The benchmark is heterogeneous across categories: evaluation tracks are label-balanced, while category frequencies remain uneven because policy-discriminative data availability differs by category.The category-wise statistics omit sft_think because it is distribution-matched to sft.
  • B.4 Policy coverage: The canonical benchmark contains 28 policy variants—16 Adaptive and 12 held-out Shift policies—while BP-Adapt adds 14 training-only augmentation policies, two per category.Augmentation policies never appear in Adaptive or Shift evaluation splits.
  • B.5 Attribute-vote statistics: The canonical three-model voted metadata contains 8,973 image records and 475,569 attribute decisions, with most decisions unanimous and remaining accepted cases resolved by two-out-of-three majority vote.No unresolved attribute decisions enter the current voted output, and the reported rates are attribute-decision rates rather than image-level rates.
  • C Policy and Rule Catalog: The policy catalog represents each category with atomic image attributes, distinguishes triggers from contextual exemptions, and assigns ADAPTIVE, SHIFT, or AUG. branch tags to executable policy rules.ADAPTIVE policies support supervised training and Adaptive evaluation; SHIFT policies test policy generalization; AUG. policies create training-only BP-Adapt boundary pairs.

C.1 Training-only policy extension … E.1 Policy Shift Score

The training extension prevents memorizing fixed policy-answer mappings by using arbitrary policy text and matched boundary groups. The supplied rules define policy-specific block/pass behavior across nudity, violence, and regulated-substance examples, while reasoning supervision uses concise fixed-label policy traces.

  • C.1 Training-only policy extension: Training-only policies cover all seven risk categories and include textual policy rephrasings to force reading arbitrary policy text.These policies are excluded from the 28 evaluation policies and the Adaptive or Shift splits.
  • C.1 Training-only policy extension: 1,908 boundary groups yield 3,816 balanced training rows pairing the same image and risk category with blocking and passing policies.The main post-training recipe retains only matched boundary groups, balanced between block and pass labels.
  • C.2 Category 01. Nudity, Sexual Content & Fetish (Nudity & Bodily Exposure Nuances): The nudity category separates triggers such as swimwear, exposed genitalia, sexual acts, deep kissing, disturbing tissue, and fetish gear from contextual exemptions.Medical, artistic, and cultural or sport contexts can suppress or narrow blocking under selected policies.
  • Policies and executable rules: Nudity policies vary from broad blocking under Strict Puritan / Family Safe to conditional allowances under Medical & Educational / Scientific Reference.Policy B blocks exposed genitalia, swimwear, suggestive actions, deep kissing, and cultural or sport nudity, whereas Policy C exempts medical or educational exposure.
  • C.3 Category 02. Violence, Hate, Terrorism & Self-Harm: The violence category distinguishes real and fictional weapons or graphic violence, animal cruelty, extremism, self-harm, and eating-disorder promotion, with news, combat-sport, and educational exemptions.These attributes support policy-specific treatment of real-world harm, fictional content, and contextual exceptions.
  • Policies and executable rules: Violence policies range from total blocking under Zero Tolerance / School & Kids to permitting fictional graphic violence under Gaming & Creative Platform / Fiction Only.The gaming policy blocks real graphic violence, animal cruelty, extremism, eating-disorder promotion, and self-harm when fictional framing is absent.
  • C.4 Category 03. Regulated Goods & Substances: The regulated-goods category identifies tobacco, cannabis, hard drugs, medical drugs, alcohol, laboratory or educational samples, and active consumption.Consumption is distinguished from static product display by a person performing the intake action.
  • Policies and executable rules: Substance policies differ by context: Mainstream Social blocks hard drugs, cannabis, and tobacco, whereas Regional Permissive blocks only hard drugs and Retail & Pharmacy blocks hard drugs or consumption acts.The latter policies allow static displays and educational or pharmaceutical contexts under their stated rules.

E.2 Main results with rerun variation · E.3 Per-category model accuracy

The paper measures fixed-condition rerun variability for POLICYSHIFTBENCH and reports category-wise decision accuracy across adaptive and shifted splits. Rerun statistics isolate evaluation-run variability, while category results summarize performance across seven safety domains.

  • E.2 Main results with rerun variation: E.2 evaluates the same checkpoint repeatedly on a fixed test split with unchanged prompts, decoding, parsing, and evaluation scripts.The measured variation reflects inference-backend and runtime variability, not split construction, initialization, training randomness, or annotation variability.
  • E.2 Main results with rerun variation: Each rerun metric reports the corresponding main-table value with one-sigma sample standard deviation over completed repeat runs.The standard deviation is computed for each metric value mi from completed repeats.
  • E.2 Main results with rerun variation: The rerun analysis reports standard deviation rather than standard error or confidence intervals and excludes models without repeated evaluations.It makes no assumption that evaluation errors are normally distributed.
  • E.3 Per-category model accuracy: Tables 12 and 13 report category-wise decision accuracy for every model in the main table using the parsed outputs from Table 2.The reported values are percentages for the Adaptive Split id_test branch and Shift Split ood_test branch.
  • E.3 Per-category model accuracy: The seven categories are Nudity and sexual content (NR), Violence/hate/self-harm (VH), Regulated goods (RG), IP/brand safety (IP), Cultural/religious sensitivity (CR), Privacy/PII (PII), and Text-in-image safety (TXT).These abbreviations define the category breakdown used in the accuracy tables.
  • E.3 Per-category model accuracy: Mean is the unweighted average over the seven category accuracies.This aggregation applies to the category-wise accuracy results.

F Additional Implementation Details … G.6 Human audit details

The implementation details define matched-policy training and a reasoning-based diagnostic, while quality control and human audits distinguish policy-conditioned decision accuracy from benchmark-label quality. Results show that concise no-think BP-Adapt is substantially stronger and faster than the think-mode alternative, and that policy-conditioned moderation remains difficult even for humans.

  • F Additional Implementation Details: PolicyShiftGuard uses Qwen2.5-VL 3B and 7B backbones, with BP-Adapt as the second-stage recipe for final models and Think PRM-RL retained diagnostically.The main models use Qwen2.5-VL 3B and 7B; Think PRM-RL is not the main recipe.
  • F Additional Implementation Details: BP-Adapt trains matched boundary pairs sharing an image and risk category, with one blocking policy and one passing policy.Blocked targets are true | XX, while passing targets are false; separate losses supervise answers, labels, and blocked categories.
  • F.1 Think PRM-RL diagnostic: Think PRM-RL uses format-gated and answer-gated rewards, then applies process shaping for image grounding, policy use, and reasoning consistency.Fully correct answers receive process terms rvis, rpolicy, and rlogic, while incorrect labels or categories receive negative rewards.
  • F.1 Think PRM-RL diagnostic: 64.6 Avg. F1 is achieved by the 7B think-mode stage-2 variant, versus 76.9 Avg. F1 for no-think BP-Adapt.The think-mode branch is slower because it generates and judges rationales, so it is used as an ablation rather than the main recipe.
  • G.1 Automatic quality control: Quality control combines multi-model majority voting with split hygiene, image-disjoint major partitions, and validator-backed mini subsets.These checks reduce single-model annotation noise and support reproducible construction and spot checking.
  • G.2 Human auditing protocol / G Quality Control and Human Audit: Human auditing separates blind model-evaluation decisions from data-quality judgments that reveal the gold answer and assess label reasonableness.Human Performance measures blind accuracy, whereas Qualified Rate measures whether annotators consider labels reasonable.
  • G.3 Human performance prompt / G.4 Label-quality audit prompt / G.6 Human audit details: Human prompts require judging only visible image evidence under the active policy, with model-eval requiring a violation decision and category selection, and label audits checking qualification and issue types.Auditors may flag wrong labels, wrong categories, policy ambiguity, image issues, or prompt/format problems.
  • G.5 Interpretation / G.6 Human audit details: The human-performance versus qualified-rate gap shows that policy-conditioned moderation is nontrivial even when benchmark labels are reasonable.The audit interface records decisions, categories or quality flags, notes, timing, sample metadata, and timestamps without collecting demographic or sensitive personal information.

H Qualitative Results

Qualitative policy-flip cases hold image evidence constant while changing only the active policy, exposing whether models follow policy boundaries rather than image-level safety priors. PolicyShiftGuard usually adapts its binary decision, whereas baselines often preserve decisions or misidentify the violated category.

  • Evaluation Setup: Figures 6–15 evaluate policy flips by fixing the image and risk category, changing only the active policy, and requiring the correct binary decision plus an accepted category identifier for unsafe samples.The gold answer flips from false to true across each policy pair.
  • Model Behavior: Baselines often recognize salient risk cues but keep the same decision across policy branches or assign violations to generic categories, while PolicyShiftGuard usually changes its decision when policies change.The evaluated flip isolates the displayed policy boundary even when images contain multiple safety cues.
  • Policy Categories: The qualitative cases span nudity, violence, regulated goods, IP and brand safety, cultural and religious sensitivity, and privacy or PII policy boundaries.Examples include context-sensitive versus family-safe nudity, contextual violence, educational regulated-good presentation, held-out IP policies, culturally specific rules, and privacy-specific category 06 blocking.

I Compute Resources · J Ethics and Responsible Release

The study used an internal A100-based SLURM setup for open-source training and evaluation, while responsible release addresses sensitive safety content through warnings, usage terms, auditing, and constrained outputs. The benchmark is intended strictly for evaluating and improving policy-adaptive guardrails, not as a sole automatic safety authority.

  • I Compute Resources: Open-source training and local evaluation ran on an internal SLURM cluster with NVIDIA A100 80GB GPUs.Training used four GPUs from an eight-GPU node, while local evaluation used one GPU per model checkpoint.
  • I Compute Resources: Representative runtimes came from local SLURM logs and excluded queueing, model transfers, and open-source synchronization.The compute summary covers the reported open-source training and evaluation pipeline.
  • I Compute Resources: API-based closed-source evaluations were excluded from the reported GPU totals.The compute-resource table summarizes only the open-source training and evaluation pipeline.
  • J Ethics and Responsible Release: The benchmark necessarily contains sensitive content involving nudity, violence, regulated goods, identity information, cultural or religious sensitivity, and harmful text.The resource is intended to evaluate and improve policy-adaptive guardrails, but misuse could probe moderation boundaries or overfit to a fixed policy catalog.
  • J Ethics and Responsible Release: Sensitive visual concepts and harmful text are mitigated through content warnings, acceptable-use terms, compact moderation labels, policy rules, and documentation.The release is framed strictly for safety evaluation and guardrail research.
  • J Ethics and Responsible Release: Policy labels use explicit rules and voted attributes, enabling released examples to be audited rather than treated as opaque safety labels.Model outputs are restricted to a compact moderation decision and violated category, excluding generative instructions or detailed harmful content.
  • J Ethics and Responsible Release: Release materials should include the policy catalog, rule code, evaluation scripts, model cards, dataset documentation, and content warnings.These materials support responsible use of the safety-evaluation resource.

J.1 Data and code availability · J.2 Croissant and RAI metadata · J.3 LLM usage disclosure

The paper releases its code, dataset, and supporting reproducibility materials, including validated Croissant metadata documenting data provenance and responsible-use information. It also discloses the use of LLMs and MLLMs in dataset construction, metadata curation, reasoning-trace generation, and manuscript preparation.

  • J.1 Data and code availability: The code is publicly available through the PolicyShiftGuard GitHub repository.The release supports research evaluation and reproducibility under stated responsible-use constraints.
  • J.1 Data and code availability: The PolicyShiftBench dataset is publicly available through its Hugging Face repository.The release includes benchmark and training splits, policy rules, documentation, evaluation utilities, license files, terms files, and Croissant metadata.
  • J.1 Data and code availability: The release package includes benchmark splits, training splits, policy rules, documentation, evaluation utilities, licensing materials, and reproducibility resources.These materials are intended for research evaluation under the stated responsible-use constraints.
  • J.2 Croissant and RAI metadata: The dataset release is accompanied by a validated Croissant metadata file documenting public resources, schemas, checksums, and provenance.The metadata also specifies required Responsible AI fields.
  • J.2 Croissant and RAI metadata: The metadata records dataset limitations, known biases, sensitive-information considerations, intended and out-of-scope uses, social impact, synthetic-data status, source datasets, and generation activities.These fields provide structured responsible-use context for the released data.
  • J.3 LLM usage disclosure: LLMs and MLLMs supported dataset construction and manuscript preparation, including multimodal attribute prediction, reasoning-trace generation, and writing assistance.Metadata predictions were aggregated by field-level majority vote and were not used as direct policy labels; reasoning traces were conditioned on fixed gold labels and policy rules.
Loading 2607.05910v1…