PaperlayerView exact revision on arXiv

Source-linked AI summary

Concept-Level Risk and Calibration for Governance in Diffusion Foundation Models

Kun Xu, Yushu Zhang, Tao Wang, Shuren Qi, Barbara Carminati, Elena Ferrari, Yuming Fang

arXiv:2609.08517v1cs.MM

TL;DR

Diffusion safety evaluations provide limited joint comparison across models, conditioning channels, and deployment conditions. CLRC defines concept-level probabilistic risks across those configurations and applies held-out post-hoc calibration. Its experiments find non-uniform risk patterns and systematic miscalibration, while the calibration design does not establish transfer from standard-only training to shifted protocols.

  • Problem

    Existing diffusion evaluations mainly use heuristic audits, adversarial probing, or task-specific benchmarks, with limited systematic comparison across models, channels, and deployment conditions.

    Source: §1 · p. 2

  • Method

    CLRC models governance-relevant behaviors as Bernoulli concept events, indexes risk profiles by configuration, and applies sample-level post-hoc calibration with configuration-level aggregation.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Experiments reveal structured concept-risk patterns, cross-channel shifts, and systematic miscalibration, with nominal fixed-judge reproduction frequencies higher for embedding than prompt conditioning.

    Source: §1 · p. 2, §5.2 · p. 6

  • Takeaways & Limitations

    CLRC provides a common schema for probabilistic, decision-aware comparison of concept risks across models, channels, protocols, and interventions.

    Source: §1 · p. 2

    Held-out shifted-protocol slices use maps trained on mixed-protocol development data, so they do not demonstrate transfer from a standard-only calibrator.

    Source: §5.4 · p. 7

Abstract

from arXiv · show

Diffusion models have become a core paradigm for multimedia generation, offering powerful concept-driven controllability for personalization, semantic editing, and selective unlearning. However, as semantic control extends beyond natural-language prompts to learned embeddings and intervention pipelines, the safety and governance of these systems become increasingly difficult to evaluate in a unified manner, especially for safety-sensitive, identity-linked, and other privacy-relevant concepts. Existing studies mainly rely on heuristic audits, adversarial probing, or task-specific erasure benchmarks, and therefore provide limited support for systematic comparison across models, conditioning channels, and deployment conditions. We present a concept-level probabilistic audit and reporting framework for diffusion models. We formalize governance-relevant concept behaviors as Bernoulli semantic events induced by stochastic generation, and define a Concept Risk Operator that maps model-channel configurations to structured risk profiles, enabling comparison across prompting interfaces, learned embedding channels, models, and recorded conditions. We apply sample-level post-hoc calibration and configuration-level risk aggregation, and show that probability error can change thresholded actions near policy boundaries. Experiments on SD1.5, SD2.1, and SDXL reveal consistent yet non-uniform operational risk patterns across concept families, channels, recorded conditions, and shifted protocols. In particular, embedding-based access and obfuscated prompts expose risks often understated by standard-prompt evaluation. A pooled multi-protocol calibrator improves held-out probability reliability, but we do not claim transfer from a standard-only calibrator. CLRC provides a common audit schema for probabilistic and decision-aware governance of multimedia generation systems.

1 Introduction

Concept-driven controllability creates governance challenges because sensitive concepts may persist across conditioning channels, while existing evaluations offer limited joint comparison. CLRC frames these behaviors as probabilistic events and audits risk across configurations, with attention to calibration and threshold decisions.

1 Introduction

  • Sensitive concepts may remain recoverable through alternate channels or adaptive prompts, especially in identity-linked and personalized generation.Prompt-only robustness defenses do not provide a holistic view across models, conditioning channels, and deployment configurations.Source: §1 · p. 2
  • Existing work covers erasure, red-teaming, and concept-removal benchmarks, but joint reporting across models, channels, protocols, interventions, benign loss, and calibration remains less standardized.The paper frames this as a gap in systematic comparison across deployment conditions.Source: §1 · p. 2
  • The paper asks how risk migrates across pathways, whether reproduction and bypass can be modeled as Bernoulli events, and how calibration affects threshold actions.Spillover is framed as a population-level collateral-risk diagnostic.Source: §1 · p. 2
  • CLRC is an audit and reporting schema, not a new concept detector, erasure algorithm, or calibration method.It indexes event-frequency estimates across concepts, models, channels, protocol distributions, and intervention states, and pairs them with held-out post-hoc calibration.Source: §1 · p. 2
  • The controlled protocol reveals structured concept-risk patterns, cross-channel shifts, and systematic miscalibration, with probability error changing actions near policy boundaries.The schema supports comparisons across prompt and embedding access, models, and recorded pre- and post-condition settings.Source: §1 · p. 2

2 Related Work

Prior work develops concept conditioning, editing, erasure, and safety evaluation, but these strands often address distinct tasks or attack surfaces. CLRC adds a joint reporting layer for comparing risk across models, channels, protocols, and interventions, with held-out calibration and action-sensitivity analysis.

2.1 Concept Conditioning, Editing, and Erasure in Diffusion Models

  • Personalization and editing methods inject, bind, or manipulate concepts through learned tokens, model fine-tuning, attention control, inversion, instructions, and spatial conditions.The examples include Textual Inversion [13], DreamBooth [48], Prompt-to-Prompt, Null-text inversion [38], InstructPix2Pix [1], and ControlNet [64].Source: §2.1 · pp. 2–3
  • Erasure and unlearning methods suppress concepts through inference-time interventions or model edits, while analyses report that removal can remain brittle under adversarial prompts or alternate channels.The cited approaches span Safe Latent Diffusion [51], ESD [14], and UCE [15]; Ring-A-Bell reports model-agnostic bypass behavior [56].Source: §2.1 · p. 3
  • Embedding-space attacks and defenses can instantiate interventions, making CLRC a complementary audit rather than a competing optimizer.CLRC audits reproduction, bypass, benign loss, and calibration under matched protocols.Source: §2.1 · p. 3

2.2 Safety and Governance Challenges in Diffusion Models

  • Safety benchmarks address valuable but task-specific targets; CLRC jointly indexes reproduction, bypass, and benign loss by model, channel, protocol, and intervention.It adds held-out human-referenced calibration and action-sensitivity analysis, claiming broader joint reporting rather than superior detector or erasure performance.Source: §2.2 · p. 3
  • Privacy, memorization, fairness, and governance concerns coexist with evaluations that often rely on heuristic audits or aggregate rates rather than calibrated Bernoulli concept events.The cited privacy evidence includes verbatim training-image leakage [4] and membership-inference risks [53].Source: §2.2 · p. 3

3 Concept Risk Modeling and Calibration

The framework treats governance-relevant generation outcomes as stochastic concept events and organizes their risk across model, channel, protocol, and intervention configurations. It then calibrates sample-level probabilities against human references and aggregates them for configuration-level analysis.

3.1 Concept Event Family

  • A conditional stochastic generator maps model, channel, noise, and conditioning input to an image distribution, with governance conditions replacing the baseline pipeline.Human-reference events are archived binary annotations, while operational events use a fixed judge and threshold; the reference is not latent truth or an oracle.Source: §3.1 · pp. 3–4
  • Benign utility loss is measured on a separate 20-concept auxiliary benign set, and its population-level diagnostics are excluded from the target risk tensor.The protocol uses a tolerance of 0.05 for thresholding the benign-loss rate.Source: §3.1 · p. 4

3.2 Structured Risk Profiling

  • Structured risk profiles collect admissible reproduction and bypass event probabilities across concepts, event types, models, channels, protocol distributions, and conditions.Spillover is treated separately as a distribution-level diagnostic.Source: §3.2 · p. 4
  • Signed channel and model risk gaps compare event frequencies while holding the other configuration axes fixed.Under configuration-invariant judge error with α_k + β_k < 1, the gaps preserve human-reference ordering; configuration-dependent error need not.Source: §3.2 · pp. 4–5

3.3 Probabilistic Estimation and Calibration

  • Calibration maps a clipped score proxy to human-reference probabilities at the sample level, then aggregates calibrated probabilities by concept and configuration.Development data are pooled across concepts and protocol slices, while aggregation retains the concept index; pooled ECE and Brier values are sample-level diagnostics, not concept-specific policy risks.Source: §3.3 · p. 5
  • Held-out probability reliability is evaluated against human-labeled score–label pairs, while raw-versus-calibrated action disagreement measures sensitivity rather than decision accuracy.The calibration target is the human-reference event, distinct from the operational judge event.Source: §3.3 · p. 5

4 Probabilistic Semantic Risk Theory

The framework treats concept-event risk as a probability under an intervention-transformed generation distribution. Comparative operators support model and channel comparisons, while their interpretation depends on judge-error conditions and distributional stability bounds.

4.1 Event Tensor and Intervention-Induced Risk

  • Interventions transform the generation distribution, and each concept-event risk is its probability under the resulting distribution.This makes semantic risk a distributional property rather than a prompt-specific artifact.Source: §4.1 · p. 5
  • Total variation distance upper-bounds the change in any single concept-event probability between two distributions.A refined bound separates distributional shift from semantic boundary mass around a threshold.Source: §4.1 · p. 5

4.2 Comparative Risk Operators and Conditional Robustness

  • Observed comparative gaps preserve human-reference ordering only under configuration-independent judge error with false-positive plus false-negative rates below one.With configuration-varying error, gap bias can differ between configurations, so comparative invariance is conditional rather than guaranteed.Source: §4.2 · pp. 5–6
  • Comparative semantic risk is bounded by total variation between induced distributions, but adaptive judge evasion can make observed ordering unreliable.Post-hoc calibration and sampling intervals do not certify robustness to adaptive judge evasion.Source: §4.2 · p. 6

5 Experiments

Across the evaluated models and protocols, operational risk varies by concept family, conditioning channel, recorded condition, and protocol. Held-out calibration improves probability reliability, while probability errors can alter actions near policy thresholds; the reported calibration does not establish transfer from standard-only training.

5.1 Experimental Setting

  • The study evaluates SD v1.5, SD v2.1, and SDXL across prompt and textual-inversion channels, 64 core concepts, and standard, shifted, and obfuscated protocols.Each evaluation cell contains 50 images total; the recorded post-condition is opaque and has no component-level effect attribution.Source: §5.1 · p. 6

5.2 Structured Concept Risk Across Models and Conditioning Channels

  • SDXL remains above the benign baseline, and the 90th-percentile risk exceeds the median in every model–channel configuration, most strongly under embedding conditioning.Category means can conceal particularly vulnerable concepts.Source: Figure 2 · p. 6, §5.2 · p. 7

5.3 Comparative Risk Gaps, Residual Reproduction, Bypass, and Semantic Spillover

  • Mean target-family risk is 0.32 for standard prompts versus 0.58 for embeddings; bypass is 0.11 under standard versus 0.19 under obfuscated prompts, while archived post-condition residual reproduction remains non-zero.These are complementary slices, not a comparison of accuracy or a new safety algorithm.Source: Table 2 · p. 7, Figure 3 · p. 7, §5.3 · p. 7
  • Under the archived post-condition, residual reproduction spans 0.13–0.16 for prompts and 0.26–0.33 for embeddings; benign loss is higher under obfuscation for every displayed backbone.Benign-loss values average the disjoint 20-concept auxiliary benign set, not a target family.Source: Table 2 · p. 7, Figure 3 · p. 7, §5.3 · p. 7

5.4 Held-out Calibration and Action Sensitivity

  • The channel-level calibration maps pool mixed-protocol development data, so held-out results do not demonstrate transfer from standard to shifted data.The authors note that score mappings may shift with style or obfuscation.Source: §5.4 · p. 7
  • Across six held-out slices, isotonic mapping reduces ECE and Brier error throughout, while raw and calibrated mappings disagree on actions near intermediate thresholds.The largest displayed raw errors occur in embedding and obfuscated slices; the comparisons are descriptive, not causal.Source: §5.4 · p. 7, Figure 4 · p. 8, Table 3 · p. 8

A.1 Calibration, Observation Noise, and Governance Bounds

The section distinguishes latent human-reference events from thresholded judge events and models how symmetric judge noise affects observed risk. It also relates calibration error to thresholded actions while cautioning that action disagreement is sensitivity, not correctness.

A.1 Calibration, Observation Noise, and Governance Bounds

  • The latent semantic event is represented by a human-reference annotation, while the judge event is separately thresholded under a conditionally symmetric noise model.The model assumes 0 ≤ η_k < 1/2; it does not establish that η_k is constant across configurations.Source: §A.1 · p. 10
  • Concept-dependent judge noise can systematically distort absolute risk, and seed-level confidence intervals do not verify constant noise across configurations.The stated noise model is conditional; the intervals quantify sampling uncertainty.Source: §A.1 · p. 10
  • Near a policy threshold, the bound on action disagreement becomes vacuous without a margin condition; raw-versus-calibrated action disagreement measures sensitivity, not correctness.The compared actions threshold the raw and calibrated sample probabilities, respectively.Source: §A.1 · pp. 10–11

A.2 Structured Stability and Judge-Noise Robustness

The appendix gives conditional stability results for concept risks under distribution shift and judge noise, then specifies the protocol and calibration choices used for reproducible comparisons. Its claims are bounded by assumptions about the judge and the archived post-condition.

A.2 Structured Stability and Judge-Noise Robustness

  • Borderline generations are less stable because small distribution shifts can flip thresholded semantic events; the bound separates distribution shift from mass near the semantic boundary.The boundary region is |f_k(x) − τ_k| ≤ ρ.Source: §A.2 · p. 11
  • Under configuration-independent class-conditional noise, comparative risk-difference signs are preserved when 1 − α_k − β_k > 0.The result is conditional and does not establish configuration-invariant judge error.Source: §A.2 · p. 11

B.1 Reproducible Protocol

  • The reproducible protocol uses 64 core concepts across four families, with 50 generated images per configuration cell and separate development and held-out core partitions.The core held-out partition evaluates reliability, ECE, and Brier; the development partition fits thresholds and channel-level calibrators.Source: §B.1 · p. 11
  • Prompt–embedding gaps compare declared deployment distributions rather than a causal descriptor-to-token substitution, and πpost permits no component-level attribution.πpost is an opaque, indivisible archive-stable condition key.Source: §B.1 · p. 11
  • Channel-level calibrators pool development pairs across concepts and available protocol slices, so shifted and obfuscated rows assess a pooled multi-protocol calibrator, not standard-only transfer.Isotonic regression is the primary monotone fit; Platt scaling is the parametric baseline.Source: §B.1 · pp. 11–12

B.2 Experimental Setup and Evaluation Protocols

The evaluation defines operational event frequencies and calibration procedures for fixed-judge audits, with explicit boundaries on what those results establish. Under the matched standard protocol, embedding frequencies exceed prompt frequencies for target families, while both channels decline from SD1.5 to SDXL.

B.2 Experimental Setup and Evaluation Protocols

  • Operational risk is the empirical frequency of thresholded fixed-judge events; reproduction, standard bypass, obfuscated bypass, and spillover use distinct declared distributions or diagnostics.Spillover is a population-level benign-risk diagnostic, not a sample-level event type.Source: §B.2 · p. 12
  • Core concepts contribute 3,840 labeled images, split concept–channel-stratified 70/30; development fits thresholds and pooled channel calibrators, while held-out data evaluates reliability.The 20 auxiliary benign concepts contribute labels only for spillover-event thresholds and are excluded from channel calibrators and Table 3.Source: §B.2 · p. 12
  • Shifted and obfuscated protocols stress the generator and intervention pipeline, not judge-aware evasion; fixed-judge risks do not certify robustness across configurations or against another judge.A judge-aware attacker may seek human-positive outputs missed by the operational judge.Source: §B.2 · p. 12
  • The reported models are SD1.5, SD2.1, and SDXL; unavailable axes in closed APIs should be marked unavailable rather than assigned zero risk.Results are scoped to named aggregate slices and the opaque πpost condition, without component-level intervention attribution.Source: §B.2 · p. 12

B.3 Audit Counts and Descriptive Marginal Intervals

  • Embedding frequencies exceed prompt frequencies, and both channels decrease from SD1.5 to SDXL; target-family gaps are 0.257, 0.247, and 0.240 respectively, versus 0.030 for core-benign.The target-family gaps correspond to SD1.5, SD2.1, and SDXL; they are point differences under the matched standard protocol.Source: §B.3 · pp. 12–13, Table 4 · p. 13
  • Wilson intervals are pooled-image descriptors, not concept-clustered or matched-seed paired contrast intervals; entries remain conditional on the fixed judge.The passage recommends concept-clustered, matched-seed bootstrap inference for confirmatory contrasts.Source: §B.3 · p. 13, Table 4 · p. 13

B.4 Threshold-Selection Ablation for Concept Event Construction

The ablation compares the default F1-max event threshold with a conservative development-FPR rule. The prompt-to-embedding operational gap persists across the representative slices, while event-threshold changes do not alter human-reference calibration.

B.4 Threshold-Selection Ablation for Concept Event Construction

  • The ablation compares F1-max thresholds with a conservative rule targeting development FPR = 0.05.The default rule selects concept-event thresholds on the threshold-fitting split.Source: §B.4 · p. 13
  • Embedding operational frequency exceeds prompt frequency in all three displayed slices, and the channel gap’s sign remains unchanged under the alternative threshold rule.The reported gaps change from 0.25 to 0.24 for SD1.5-Std and remain 0.22 for SD2.1-Shf and 0.23 for SDXL-Obf.Source: §B.4 · p. 13
  • Changing the event threshold does not change the proxy, human-reference label, or calibration metrics, so no calibration change is attributed to this ablation.The passage names r, Y, q, ECE, and Brier as unchanged by changing τ_k.Source: §B.4 · p. 13

B.5 Threshold-Scan Summary for Calibration-Induced Action Sensitivity

The threshold scan summarizes calibration-induced changes to sample actions across representative settings. Raw-versus-Isotonic disagreement is larger than Raw-versus-Platt in the displayed slices, with peak disagreement at intermediate thresholds.

B.5 Threshold-Scan Summary for Calibration-Induced Action Sensitivity

  • The scan is summarized by disagreement-curve AUC, peak disagreement, and the threshold where that peak occurs.Figure 8 reports these quantities for Raw-vs-Isotonic and Raw-vs-Platt comparisons.Source: §B.5 · p. 13, Figure 8 · p. 14
  • Raw-vs-Isotonic action-disagreement AUC exceeds Raw-vs-Platt across representative settings, with the largest displayed AUC values in embedding-based and obfuscated slices.These are descriptive comparisons, not causal attributions to channel or protocol.Source: §B.5 · p. 13, Figure 8 · p. 14
  • Raw-vs-Isotonic peak action disagreement reaches roughly 0.17 in the largest displayed slice, while the smallest peak is SDXL-P-Std and the largest is SD2.1-E-Obf.The statistic measures action changes relative to raw scores, not accuracy or regret.Source: §B.5 · p. 13, Figure 8 · p. 14
  • Peak action disagreement occurs at intermediate thresholds, approximately 0.45–0.60, rather than near either extreme.The passage attributes this pattern to more samples lying near the decision boundary around the center of the score distribution.Source: §B.5 · p. 13, Figure 8 · p. 14

C Governance Implications

The governance implications call for concept-resolved, interface-aware risk reporting rather than a single global safety score. They also emphasize shifted protocols, intervention trade-offs, and calibrated probabilities for policy decisions.

C Governance Implications

  • Governance should evaluate identifiable semantic events rather than rely on a global safety score, because average safety can conceal exposure on governance-critical concepts.The paper recommends concept-resolved auditing across concept families, channels, model architectures, and protocol distributions.Source: C Governance Implications · p. 14
  • Risk reports should distinguish conditioning channels, protocol distributions, and model variants because newer checkpoints reduce some risks without eliminating them uniformly across access paths.Prompt-only benchmarks and aggregate claims about a model family are not sufficient for these comparisons.Source: C Governance Implications · p. 14
  • Intervention audits should jointly report target-family suppression, shifted or indirect bypass, and spillover on benign controls to make harm-reduction and capability-loss trade-offs explicit.A reduction in average target risk does not by itself establish intervention success under obfuscated bypass or benign-fidelity loss.Source: C Governance Implications · pp. 14–15
  • Shifted and obfuscated protocols should be part of routine evaluation because concept realization and intervention robustness degrade under those conditions.Standard-prompt-only conclusions are described as likely overly optimistic.Source: C Governance Implications · p. 15
  • Policy thresholds should use calibrated concept-risk probabilities, with reliability checked on nominal and stress-test distributions before decisions are applied.The paper separates scoring, calibration, and decision-making as distinct stages.Source: C Governance Implications · p. 15
  • The proposed workflow estimates configuration-specific risks, fits calibration on development annotations, verifies held-out reliability, and then applies predeclared thresholds or intervention rules.Any change to the model, judge, channel, intervention, or deployment protocol triggers renewed validation rather than automatic transfer of the prior report.Source: C Governance Implications · p. 15

Related papers

Loading 2609.08517v1…