PaperlayerView exact revision on arXiv

Source-linked AI summary

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

Jiacheng Zhang, Haoyu He, Sen Zhang, Shen Wang, Xiaolei Xu, Yuhao Sun, Meng Shen, Feng Liu

arXiv:2606.29887v1cs.AI

TL;DR

Application-specific safety policies are increasingly needed, but evaluating whether guardrails can execute them in context remains challenging. SafePyramid benchmarks rule understanding, dependency reasoning, and adaptation to novel policy frameworks, finding that GPT-5.5 exactly matches all violated rules in only 54.0%, 35.3%, and 12.9% of L0, L1, and L2 cases.

  • Problem

    Evaluating whether guardrails can reliably execute application-specific safety policies in context remains largely unexplored and challenging.

  • Method

    SafePyramid evaluates violated-rule set prediction across three levels covering individual-rule understanding, rule dependencies, and novel policy-framework adaptation.

  • Results

    54.0%, 35.3%, and 12.9%: GPT-5.5 exactly matched the full violated-rule set on L0, L1, and L2, respectively.

  • Takeaways & Limitations

    Current systems remain far from reliable, especially under rule dependencies and newly defined policy concepts.

  • Takeaways & Limitations

    SafePyramid does not include a human performance baseline, so it does not directly measure the gap between models and expert policy auditors.

Abstract

from arXiv · show

In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies, rather than relying on predefined risk taxonomies. In this work, we study this setting under the paradigm of in-context policy guardrailing, where guardrails predict safety violations based on policy specifications provided in context. To systematically evaluate this capability, we introduce SafePyramid, a safety benchmark comprising 1,000 multi-turn conversations across 10 domains and 3,000 corresponding application-specific policies, which together contain 61,699 distinct natural-language rules. SafePyramid organizes the evaluation into three difficulty levels: L0 evaluates individual-rule understanding, L1 evaluates reasoning over rule dependencies, and L2 evaluates adaptation of full novel policy frameworks defined in context. To ensure benchmark quality, we employ a rigorous multi-stage pipeline to construct and validate the benchmark. Using SafePyramid, we evaluate 10 frontier LLMs and 5 policy-configurable guardrails and find that in-context policy guardrailing remains highly challenging: even the best-performing model, GPT-5.5, exactly identifies the full set of violated rules in only 54.0%, 35.3%, and 12.9% cases on L0, L1, and L2, respectively. These results highlight the limitations of current guardrails and call for stronger in-context policy guardrails that can reliably execute policies, resolve rule dependencies, and adapt to novel policy frameworks.

1 Introduction

SafePyramid benchmarks in-context policy guardrailing for application-specific safety policies through a three-level hierarchy targeting rule understanding, dependency resolution, and adaptation to novel policy frameworks. Across 1,000 multi-turn conversations, even GPT-5.5 exactly identifies all violated rules in only 54.0%, 35.3%, and 12.9% of L0, L1, and L2 cases, respectively.

  • Motivation: Application-specific risk categories and safety standards vary substantially, limiting guardrails built around fixed predefined taxonomies.Examples include youth-safety policies for short-video platforms and policies for video-gaming discussion forums.
  • Benchmark design: The hierarchy evaluates individual-rule understanding at L0, rule-dependency reasoning at L1, and adaptation to novel policy frameworks at L2.L0 includes decisive rules supported by sufficient conversational evidence and topical distractor rules that are not triggered.
  • Benchmark: SafePyramid evaluates complete violated-rule sets rather than binary safe/unsafe labels across 1,000 multi-turn conversations spanning 10 safety domains.Each conversation is paired with level-specific policy specifications.
  • Results: 54.0%, 35.3%, and 12.9% are GPT-5.5’s exact violated-rule-set match rates on L0, L1, and L2, respectively.The evaluation covers 10 frontier LLMs and 5 guard models, and performance drops monotonically across the hierarchy.
  • Results: The results indicate that in-context policy guardrailing remains far from solved, especially when models must execute increasingly complex policies supplied in context.The benchmark is motivated by the need to handle application-specific policies rather than rely on general predefined safety standards.

2 Related Work

Prior guardrails and benchmarks largely rely on fixed safety taxonomies or evaluate adjacent capabilities such as instruction following, rule reasoning, and context learning. SafePyramid instead targets policy-configurable guardrailing through violated-rule prediction and separate evaluation of rule understanding, dependency resolution, and adaptation to novel policy frameworks.

  • LLM-based guardrails: Existing guardrails typically classify inputs or outputs under predefined risk taxonomies, limiting their suitability for in-context policy guardrailing.Examples include Llama Guard, ShieldGemma, AEGIS, WildGuard, Granite Guardian, and Qwen3Guard.
  • SafePyramid’s distinction: SafePyramid uniquely formulates violated-rule set prediction as its evaluation task and separately tests individual-rule understanding, rule-dependency resolution, and adaptation to novel policy frameworks.It differs from benchmarks focused on fixed taxonomies, general constraint following, rule application, or context adaptation alone.
  • LLM-based guardrails: User-provided-policy guardrails support inference-time rules but differ substantially in policy handling and expected outputs, leaving the paradigm under-defined.The cited methods include ShieldLM, GPT-OSS-Safeguard, DynaGuard, GSPR, and FlexGuard.
  • LLM safety benchmarks: Existing safety benchmarks mainly assess harmful-request refusal, toxic or biased generations, or jailbreak resistance using fixed labels, harm categories, or annotation criteria.These benchmarks measure general safety behavior but do not directly establish application-specific policy violations.
  • Instruction following, rule-based reasoning, and context learning benchmarks: Instruction-following, rule-based reasoning, and context-learning benchmarks evaluate related capabilities from distinct perspectives, including constraint satisfaction, rule application, and adaptation.The supplied passage identifies these benchmark families as closely related to in-context policy guardrailing.

3 SafePyramid

SafePyramid evaluates in-context policy guardrailing as violated-rule set prediction under policies supplied in context. Its hierarchy isolates individual-rule understanding, rule-dependency resolution, and adaptation to fictional policy frameworks, with protocols and metrics designed to assess complete and high-fidelity rule recovery.

  • Problem setting: Guardrails must predict the subset of in-context natural-language policy rules violated by a multi-turn conversation, rather than only assigning a binary safe/unsafe label.The task supplies a policy P containing rules r_1 through r_K and evaluates the predicted violated-rule subset.
  • Hierarchical design: SafePyramid’s three levels increase difficulty from understanding individual rules, to resolving dependencies, to adapting to novel safety policy frameworks.Each level retains violated-rule identification while adding a distinct reasoning challenge.
  • L1: Rule dependencies: L1 introduces exception and conditional rules that can waive, reinterpret, or strengthen base-rule requirements, requiring context-dependent violated-rule prediction.Exceptions override default classifications under specified contexts, while conditionals impose additional safeguards when their conditions hold.
  • L2: Novel frameworks: L2 rewrites L1 policies using fictional concepts, terminology, and semantics to prevent solving through pretrained safety priors or familiar taxonomies.The rewritten policies preserve decisive, distractor, exception, and conditional rule structure while changing the governing framework.
  • Evaluation protocols and metrics: Per-policy evaluation tests complete violated-rule-set execution, while per-rule evaluation supports fixed-category or binary interfaces by aggregating one-rule classifications.Rule matching is reported at multiple thresholds because exact full-set agreement remains difficult; τ = 0.7 is used as the lowest threshold because cases average around 6 violated rules and require over 80% agreement.

4 Experiments

Experiments across frontier LLMs, configurable guardrails, and agentic harnesses show that in-context policy guardrailing remains difficult, especially for dependent rules and novel policy frameworks. Decomposition, additional reasoning, and harnesses improve performance, but gains vary by model and come with cost trade-offs.

  • Evaluation setup: 10 frontier LLMs and 5 policy-configurable guard models are evaluated for in-context policy guardrailing.The frontier models use their strongest available reasoning effort; the configurable models support customized policies, but only GPT-OSS-Safeguard models support arbitrary violated-rule set output.
  • Overall performance: 54.2% average RMR, 34.9% average RMR@1.0, and 19.7% average RDR are achieved by the best overall model, GPT-5.5.The results show that providing policies in context is insufficient for reliable policy execution.
  • Difficulty by level: Performance decreases from L0 to L2 across nearly all models, with exceptions and conditionals creating difficulty beyond independent rule classification and novel frameworks remaining especially challenging.The L2 decline indicates that adapting to novel policy frameworks is a major bottleneck.
  • Reasoning effort: 5.5 and 14.5 points are the xhigh-over-low reasoning-effort RMR gains on L1 and L2, while RDR decreases by 3.9 and 15.7 points, respectively.The gains are small on L0 and larger where models must resolve dependencies or adapt to novel frameworks.
  • Harnesses and cost: 60.4% average RMR and 40.4% RMR@1.0 are achieved by Claude-Opus-4.7 with Claude Code, while average RDR falls from 23.5% to 17.4%.For GPT-5.4, Codex and Claude Code raise average RMR from 56.0% to 58.0% and 57.9%, respectively; performance is not tightly coupled to cost.

5 Limitations and Future Work

SafePyramid’s limitations include the absence of a human performance baseline, text-only modality coverage, and finite policy coverage. Future work should evaluate expert human auditing, extend guardrailing to multimodal inputs, and broaden policies across domains, languages, jurisdictions, and real-world variants.

  • Human baseline: SafePyramid lacks a human performance baseline, so it does not directly measure the gap between model performance and expert human-level policy auditing.Benchmark labels were validated through cross-model validation and human review, but trained human policy auditors were not evaluated on the same rule-matching task.
  • Modality coverage: SafePyramid focuses on text-based user–model conversations, limiting evaluation when policy violations depend on evidence distributed across images, audio, video, or documents.Extending in-context policy guardrailing to multimodal settings is identified as an important direction for future work.
  • Policy coverage: SafePyramid’s policy coverage is finite despite spanning 10 safety domains and a large set of policy rules.Future work can add domains, languages, jurisdictions, and real-world policy variants incorporating domain-specific definitions, regional legal requirements, operational constraints, and evolving internal standards.

6 Conclusion

SafePyramid is a hierarchical benchmark for testing whether guardrails can execute safety policies provided in context. It diagnoses rule understanding, dependency resolution, and adaptation to novel policy frameworks, finding that current systems remain far from reliable.

  • Benchmark: SafePyramid evaluates whether guardrails can reliably execute safety policies provided in context.The benchmark is designed specifically for in-context policy guardrailing.
  • Diagnostic capabilities: The benchmark diagnoses three capabilities: understanding individual rules, resolving rule dependencies, and adapting to novel policy frameworks.These capabilities structure the evaluation of in-context policy guardrailing.
  • Findings: Evaluations of 10 frontier LLMs and 5 policy-configurable guard models show that current systems remain far from reliable.The conclusion reports this finding across both frontier language models and policy-configurable guard models.
  • Findings: Current systems are especially unreliable when reasoning under rule dependencies.The supplied passage identifies rule dependencies as a particularly challenging condition.

Ethics Statement

SafePyramid includes safety-sensitive conversations and policies across several high-risk domains, but uses fictionalized, controlled generation to reduce misuse risks. The benchmark is intended for research on policy-configurable guardrails and safety evaluation.

  • Risk Mitigation: SafePyramid covers fraud, defamation, privacy, and sexual-content scenarios while avoiding identifiable real individuals, organizations, and real-world incidents.Conversations are fictionalized and generated under controlled constraints, with only generic references required by scenarios.
  • Intended Use: The benchmark is intended for research on policy-configurable guardrails and safety evaluation, not for generating harmful content.

A Benchmark Example … E Instruction Prompt for Agent Harness Experiment

The supplementary sections provide benchmark examples, scenario seeds, validation checklists, and the prompts used for policy-configurable guards and agentic harness experiments. Examples span L0, L1, and L2, with L2 introducing a novel framework whose rules reference section definitions.

  • A Benchmark Example: The same conversation is reused across the L0, L1, and L2 benchmark cases.This reuse is documented in Table 6, alongside separate policy samples for each difficulty level.
  • A Benchmark Example: The L0 example provides an individual policy sample for the benchmark.Table 7 is specifically identified as the L0 policy sample.
  • A Benchmark Example: The L1 example contains 30 rules distributed across two tables.Rules 1–20 appear in Table 8, while rules 21–30 continue in Table 9.
  • A Benchmark Example: The L2 example defines a novel framework whose 32 policy rules reference definitions by section number.The framework appears in Table 10, with rules 1–32 continued across Tables 11–13.
  • B Domains and Scenarios: SafePyramid includes conversation scenario seeds for constructing benchmark cases across its domains.Table 14 presents the first of five scenario-seed tables.
  • C Validation Checklists: Validation materials separately cover conversations, L0 policies, L1 policies, and L2 frameworks and policies.Tables 19–22 provide the corresponding validation checklists.
  • D System Prompt for Policy-configurable Guards: The policy-configurable guard experiments use model-specific prompts for FlexGuard-Qwen3-8B, GPT-OSS-Safeguard-120B/20B, DynaGuard-8B, and ShieldLM-14B-Qwen.The tables state that official prompts are followed, with GPT-OSS receiving policy and output schema on the Harmony developer channel and conversations on the user channel.
  • E Instruction Prompt for Agent Harness Experiment: Every agentic harness receives a general instruction prompt through the harness CLI.This instruction is documented in Table 27.

F RMR Breakdown

This section presents RMR breakdowns for both per-policy and per-rule evaluation across six RMR thresholds.

  • Tables 28 and 29 report RMR breakdowns for per-policy and per-rule evaluation, respectively.
  • The breakdown includes RMR@1.0, RMR@0.9, RMR@0.8, RMR@0.7, RMR@0.6, and RMR@0.5.

G Per-domain Performance Breakdown · H Per-domain Effect of Agent Harness

The paper provides per-domain performance breakdowns for per-policy and per-rule evaluations across multiple models. It also examines the per-domain effect of an agent harness on per-policy evaluation using a SafePyramid subset.

  • G Per-domain Performance Breakdown: Per-policy performance across domains is reported for GPT-5.5 and Claude-Opus-4.7.
  • G Per-domain Performance Breakdown: Per-policy performance across domains is also reported for Kimi-K2.6 and DeepSeek-V4-Pro.
  • G Per-domain Performance Breakdown: Additional per-policy breakdowns cover Doubao-Seed-2.0-Pro, Gemini-3.5-Flash, Hunyuan-HY3-Preview, and Qwen-3.6-Max-Preview.The figures report these models’ performance across domains, while the text separately identifies Doubao-Seed-2.0-Pro and Hunyuan-HY3-Preview as per-policy evaluations.
  • G Per-domain Performance Breakdown: Per-policy performance across domains is further reported for GLM-5.1 and Grok-4.3.
  • G Per-domain Performance Breakdown: Per-rule performance across domains is reported for GPT-5.5, GPT-OSS-Safeguard-120B, GPT-OSS-Safeguard-20B, and FlexGuard-Qwen3-8B.These evaluations are presented in figures covering GPT-5.5 with GPT-OSS-Safeguard-120B and GPT-OSS-Safeguard-20B with FlexGuard-Qwen3-8B.
  • G Per-domain Performance Breakdown: Per-rule performance across domains is also reported for DynaGuard-8B and ShieldLM-14B-Qwen.
  • H Per-domain Effect of Agent Harness: The paper reports the per-domain effect of an agent harness for per-policy evaluation on a SafePyramid subset.

I Error Attribution

This section presents an error attribution analysis.

  • The section reports error attribution results in Figure 15.

J Root Cause Analysis

GPT-5.5’s failures primarily reflect misapplication of policy conditions, qualifiers, and exceptions. Conditional-rule false positives rise from 2% per case at L1 to 26% at L2 when novel §-clause concepts are introduced.

  • Failure mechanisms: 54% of sampled decisive-rule failures result from not applying trigger/scope predicates and instead matching surface cues.This is the dominant cause identified for GPT-5.5’s decisive-rule failures.
  • Failure mechanisms: 58% of sampled distractor-rule failures result from dropped-qualifier specificity over-read.This is the dominant cause identified for GPT-5.5’s distractor-rule failures.
  • Failure mechanisms: 43% of sampled exception-rule failures occur when an exception fires despite its activation condition being unmet.This is the dominant cause identified for GPT-5.5’s exception-rule failures.
  • L1-to-L2 conditional-rule shift: At L1, GPT-5.5 correctly treats the conditional as a modifier, applying its tightening to the base rule and listing only that base rule.The L1 evaluation uses conversation crit-605 under a policy without a regulatory framework.
  • L1-to-L2 conditional-rule shift: 26% at L2 versus 2% per case at L1 marks the conditional-rule false-positive rate when identical templates use novel §-clause concepts.Under the L2 framework for crit-605, GPT-5.5 incorrectly treats the conditional rules themselves as violations.
Loading 2606.29887v1…