PaperlayerView exact revision on arXiv

Source-linked AI summary

SaTPO: Text-to-Image Safety Alignment via Transition Preference Optimization

Ziyi Fang, Jingzhi Li, Xiaojun Jia, Xingxing Wei, Wenqi Ren, Haijun Zhang

arXiv:2610.19563v1cs.CV

TL;DR

Completed-image preferences can conflate safety with other denoising changes, complicating efforts to suppress harmful content while preserving useful generation. SaTPO aligns shared-state, one-step transitions from unsafe prompts and safe rewrites, with drift controls and benign replay. Across Stable Diffusion v1.5 and SDXL, the paper reports improved safety detection and FID over strong baselines.

  • Problem

    Completed-image preferences compress multidimensional denoising variation into coarse supervision that can entangle safety with unrelated trajectory drift, while safety and useful generation share model parameters.

    Source: §1 · p. 1

  • Method

    SaTPO uses reward-free, shared-state one-step transition preferences from unsafe prompts and safe rewrites, alongside multi-scale drift constraints and benign replay.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    Across Stable Diffusion v1.5 and SDXL, SaTPO improves broad-safety detection rates and FID over strong baselines and shows resistance to transferred adversarial attacks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The aligned model operates with standard classifier-free guidance at inference, requiring no safe rewrites or extra runtime overhead.

    Source: §1 · p. 2

Abstract

from arXiv · show

Safety alignment of text-to-image diffusion models aims to suppress harmful content without compromising useful generation capabilities. However, preferences over completed images compress multidimensional generation factors into a coarse signal, potentially driving changes beyond those required for safety. To address this, we propose Safety Transition Preference Optimization (SaTPO), which enforces step-wise alignment during denoising. SaTPO leverages the insight that an unsafe prompt and its safe rewrite naturally form a tightly coupled preference pair when conditioned on a shared parent state and noise realization. By anchoring optimization on identical parent states, SaTPO eliminates historical trajectory divergence, confining safety adaptation to an incremental alignment while preserving task-relevant semantics. Multi-scale drift constraints and benign replay further limit behavioral deviation and preserve ordinary generation. We additionally construct CoProPV, a refined dataset ensuring visually verified contrasts. Experiments on Stable Diffusion v1.5 and SDXL demonstrate improved safety-quality trade-offs. Compared with AlignGuard, PerspectiveVision unsafe-detection rates on Inappropriate Image Prompts decrease from 12.70% to 6.00% and from 15.04% to 9.30%, respectively, while FID improves from 32.28 to 25.95 and from 36.50 to 27.51. Warning: this paper includes potentially offensive content.

1 INTRODUCTION

Text-to-image safety alignment must suppress harmful outputs while preserving ordinary generation, but completed-image preferences can conflate safety with unrelated trajectory changes. SaTPO uses shared-state, reward-free transition preferences, drift controls, and benign replay; the paper reports improved safety detection and FID over strong baselines across two backbones.

1 INTRODUCTION

  • Completed-image preferences compress variation across denoising steps into a coarse signal that can entangle safety with unrelated trajectory drift and impair quality and alignment.Safety alignment is challenging because safety-related and useful behaviors share model parameters.Source: §1 · p. 1
  • SaTPO builds on the observation that an unsafe prompt and its safe rewrite form an intrinsically coupled preference pair without external reward-model ranking.Shared parent states and noise make the transition difference the rewrite-induced change in transition mean.Source: §1 · pp. 1–2
  • SaTPO converts unsafe prompts and safe rewrites into shared-state, shared-noise transition preferences, providing reward-free directional supervision for incremental, step-wise alignment.Under equal-covariance Gaussian transitions, maximizing the preference margin aligns transition-mean displacement with the rewrite-guided direction.Source: §1 · pp. 1–2
  • Multi-scale drift control combines step-wise and cross-step penalties with benign replay to limit behavioral deviation and preserve useful reference behavior.The penalties regulate individual reference-relative log-ratios and their accumulated common shift; replay anchors transitions under ordinary prompts.Source: §1 · p. 2
  • CoProPV is a visually verified, backbone-specific refinement of CoProV2 that grounds safety contrasts in each target model’s generated-image behavior.The dataset addresses mismatches between prompt-level safety specifications and image-level outcomes across diffusion backbones.Source: §1 · p. 2
  • SaTPO improves broad-safety detection rates and FID over strong baselines across Stable Diffusion v1.5 and SDXL, with reported resistance to transferred adversarial attacks.The aligned model also uses standard classifier-free guidance at inference, without safe rewrites or extra runtime overhead.Source: §1 · p. 2

2 RELATED WORK

Prior diffusion preference methods optimize completed outputs, terminal rewards, or intermediate denoising steps, while safety defenses also include prompt handling, guidance, and concept editing. SaTPO is situated among stepwise methods but constructs its preferences from unsafe prompts and safe rewrites sharing a parent state and noise.

Safe Text-to-Image Generation

  • Text-to-image safety defenses include prompt rewriting and detection, denoising-time guidance, and internal concept-erasure or model-update methods.The review also describes vulnerabilities to implicit or adversarial prompts and preference-based methods such as AlignGuard and SafetyBPO.Source: §2 · p. 3

Preference Optimization for Diffusion Models

  • Diffusion preference optimization spans terminal-reward policy gradients, reference-regularized distribution methods, and negative-preference guidance training.The review identifies DPO, DDPO, DPOK, Diffusion-DPO, DSPO, Direct Distributional Optimization, and Diffusion-NPO as examples.Source: §2 · p. 3
  • Stepwise methods compare or propagate preferences during denoising, and SaTPO forms one-step preferences from unsafe prompts and safe rewrites under shared state and noise without reward-model ranking.The review also names D3PO, SPO, LPO, RDPO, and Diffusion-SDPO as stepwise approaches.Source: §2 · p. 3

3 PRELIMINARIES

Latent diffusion denoises a Gaussian latent through reverse transitions and decodes the final latent into an image. The preliminaries introduce classifier-free guidance and shared-state stepwise preference comparisons, alongside a diagnostic showing the preference-score gap narrows late in denoising.

Denoising Transitions

  • Latent diffusion progressively denoises an initial Gaussian latent to x0, which a fixed decoder maps to a pixel-space image.For stochastic steps, the reverse transition is Gaussian with sampler-prescribed variance.Source: §3 · p. 3
  • Density-based objectives use sampling-grid steps with σt > 0, and t −1 denotes the next point on that grid.Source: §3 · p. 3

Classifier-Free Guidance

  • Classifier-free guidance combines conditional and unconditional noise predictions; the guidance scale g controls their difference, while changing the text condition changes the transition mean but not its prescribed variance.The empty condition is denoted ∅.Source: §3 · p. 3

Stepwise Preference Optimization

  • Stepwise preference optimization compares candidate successors from the same latent state under a common text condition, using transition log-ratios relative to a frozen reference model.The preference margin is the preferred candidate’s log-ratio minus the rejected candidate’s.Source: §3 · pp. 3–4

Preference-Gap Diagnostic

  • For SDXL child latents, the mean SafetyLRM score difference between preferred and rejected candidates decreases as denoising approaches completion.Figure 2 reports the trend across denoising timesteps; the diagnostic supports progressively weaker preference updates during later refinement.Source: Figure 2 · p. 4, §3 · p. 4

4 SAFETY TRANSITION PREFERENCE OPTIMIZATION

SaTPO turns unsafe prompts and safe rewrites into step-level preference supervision, using shared states and noise to align denoising transitions. Drift control and benign replay constrain adaptation while preserving generation under ordinary prompts.

4 SAFETY TRANSITION PREFERENCE OPTIMIZATION

  • SaTPO trains on coupled one-step preferences from an unsafe prompt and its safe rewrite, with shared states and noise, alongside drift control and benign replay.The procedure is reward-free and combines transition preference optimization with stepwise and trajectory-level drift control.Source: §4 · p. 4, Figure 3 · p. 5

4.1 BACKBONE-SPECIFIC SAFETY PAIR CONSTRUCTION

  • Regenerating 10,983 CoProV2 prompt pairs on SD-v1.5 and SDXL, CoProPV retains 3,351 and 3,944 pairs, respectively, with image labels matching prompt roles.Retained unsafe images must also belong to one of seven target categories.Source: §4.1 · p. 4

4.2 TRANSITION PREFERENCE LEARNING

  • Shared parent states and noise couple the safe-rewrite and unsafe-prompt successors; only the unsafe-prompt successor advances, and the rewrite-guided direction sets the preference.The method evaluates both candidates under the unsafe prompt and optimizes their transition log-probability ratios.Source: §4.2 · pp. 4–5

4.3 MULTI-SCALE DRIFT REGULARIZATION

  • Stepwise penalties limit individual candidate drift, while a trajectory-level penalty controls the signed sum of drift across the complete record.Opposite-sign increments can cancel in the final sum; each update estimates its gradient from four uniformly sampled steps.Source: §4.3 · p. 6

4.4 BENIGN REPLAY

  • Benign replay compares policy transition log-probabilities with cached reference values from COCO-caption transitions to preserve generation under ordinary prompts.Training updates only the denoiser’s LoRA parameters, and inference uses ordinary prompt-conditioned CFG sampling.Source: §4.4 · p. 6

5 EXPERIMENTS

Across SD-v1.5 and SDXL, SaTPO improves safety detection results over evaluated baselines while retaining benign-generation quality. Ablations favor benign replay, earlier preference updates, and shared parent states.

5.1 EXPERIMENTAL SETUP

  • Evaluation covers I2P, Unsafe Diffusion, and NSFW-56k for safety, and COCO captions for benign quality, with methods sharing prompts and initial noise within each backbone.The setup also compares SaTPO with existing safeguards and image-level preference methods, and reports PV, Q16, NudeNet, IP, CLIP similarity, and FID.Source: §5.1 · pp. 6–7

5.2 MAIN RESULTS

  • In selected examples, SaTPO reduces harmful content while retaining benign attributes, including SD-v1.5 portrait subjects and an SDXL aggressive interaction replaced by a standing group.Figure 4 covers seven risk categories and shares prompts and random seeds within each backbone.Source: Figure 4 · p. 7, §5.2 · p. 8
  • SaTPO has lower COCO FID than AlignGuard on both backbones and SafetyBPO on SD-v1.5, while Base has the highest CLIP similarity and lowest FID.Figure 5 compares benign-generation examples and mean CLIP similarity and FID on COCO-5000.Source: §5.2 · p. 8, Figure 5 · p. 8
  • SaTPO achieves the lowest PV, Q16, and IP detection rates on I2P and UD for both backbones; SDXL also improves all four detection metrics over adapted LPO across three safety sets.Gains extend to Q16 and IP, which were unused for data qualification, and to PV across all seven SDXL prompt categories.Source: §5.2 · p. 7, Table 1 · p. 8

5.3 ABLATION STUDIES

  • Removing benign replay lowers all four SD-v1.5 I2P detection rates but reduces COCO CLIP similarity from 0.2573 to 0.1952 and raises FID from 25.95 to 69.19.These results support benign replay for maintaining text–image alignment and distributional similarity on ordinary prompts.Source: §5.3 · pp. 8–9, Table 4 · p. 9

5.2 MAIN RESULTS

  • SaTPO achieves the lowest composite attack success rate (PV ∪ IP) across fixed adversarial-prompt cohorts on both backbones.The evaluation uses fixed prompts from MMA-Diffusion, Ring-A-Bell, and SneakyPrompt with shared seeds.Source: §5.2 · p. 8, Table 3 · p. 9

5.3 ABLATION STUDIES

  • Reversing timestep weights raises SD-v1.5 I2P IP from 5.36% to 15.94% while improving COCO CLIP similarity and FID, supporting earlier preference updates for safety.The interpretation also agrees with a decreasing mean SafetyLRM score gap across denoising.Source: §5.3 · p. 9
  • Increasing the SD-v1.5 parent-state reset interval from J = 1 to 3 or 5 raises all four I2P detection rates; shared parents outperform independently evolving dual-trajectory branches on SDXL I2P and UD PV and IP.The SDXL comparison uses matched budgets.Source: §5.3 · p. 9, Table 5 · p. 9

6 CONCLUSION

SaTPO couples one-step safety preferences through shared parent states and noise, with drift control and benign replay. Experiments report lower safety detection rates and COCO FID than evaluated safety-preference baselines, and the supplement supplies implementation and evaluation details.

6 CONCLUSION

  • SaTPO lowers I2P and UD PV, Q16, and IP detection rates and COCO FID relative to evaluated safety-preference baselines on SD-v1.5 and SDXL.The conclusion also reports safety gains on fixed adversarial prompts and ablations favoring shared parents and early denoising-step preference updates.Source: §6 · p. 9

A SUPPLEMENTARY MATERIAL

  • The supplement provides implementation details, evaluation protocols, category-level safety results, exposure-type counts, fixed adversarial-prompt comparisons, and an exploratory local-candidate diagnostic.Source: A SUPPLEMENTARY MATERIAL · p. 14

B.1 TRAINING ALGORITHM

  • Algorithm 1 collects candidates and optimizes the policy using shared-state construction, safety-pair log-ratios under unsafe prompts, and benign replay under corresponding benign prompts.Its procedure combines preference, drift, and benign-replay updates.Source: §B.1 · p. 14

B.2 DATA AND TRAINING CONFIGURATION

The configuration pairs backbone-specific CoProPV data and training schedules with separate penalties and sampling rules for preference, benign replay, and common-drift estimation. The common-drift estimator uses sampled steps to estimate the full-record penalty gradient, while transition-density conventions specify how candidates and preferences are handled.

B.2 DATA AND TRAINING CONFIGURATION

  • CoProPV retains 3,351 qualified pairs for SD-v1.5 and 3,944 for SDXL from 10,983 source prompt pairs per backbone.Pairs require a Safe label for the safe-rewrite image and an Unsafe label for the original-prompt image; no CLIP threshold or category reweighting is used.Source: §B.2 · p. 14
  • SD-v1.5 trains for two epochs on CoProPV, while SDXL trains for one epoch, with both starting from their respective base models.Training uses AdamW, and the sampling grid retains each backbone’s original noise indices.Source: §B.2 · p. 14
  • Preference minibatches retain 15 of 20 collected steps with positive w(t), while local penalties use those minibatches and common drift uses the full record.Benign replay samples cached transitions at positions {2, 7, 12, 17}; one benign prompt record supplies four transitions per update.Source: §B.2 · p. 15
  • Clipping applies to per-dimension log-ratios, so the exponentiated clipped quantity is a geometric-mean density ratio rather than the joint latent density ratio.Local and benign penalties use Φ directly, whereas the common penalty uses Φ/κcom.Source: §B.2 · p. 15
  • Reward scores do not enter preference optimization, although SDXL retains latent-reward scoring for diagnostics.SD-v1.5 constructs pairs without loading the diagnostic model.Source: §B.2 · p. 15

B.3 GRADIENT ESTIMATION FOR THE COMMON-DRIFT PENALTY

  • The common-drift estimator samples four steps uniformly without replacement, yielding the full-record penalty gradient in expectation while reducing steps that retain gradient graphs.All steps on the selected record still require forward evaluation, and the surrogate’s scalar value is not the exact penalty value.Source: §B.3 · p. 16

B.4 TRANSITION-DENSITY CONVENTIONS

  • The preference logit uses β as its scale, and candidate construction and preference assignment determine the supervision.The supplied ordering labels the candidates as w and l; the sampler’s terminal-index convention is held fixed across candidate generation and likelihood evaluation.Source: §B.4 · p. 16

C.1 DATASET CONSTRUCTION AND EVALUATION SCOPE

The evaluation combines fixed prompt sets and matched generation settings to compare safety defenses across backbones. Dataset construction specifies prompt counts and category strata, while generation settings fix each backbone’s sampler and checkpoint.

C.1 DATASET CONSTRUCTION AND EVALUATION SCOPE

  • I2P-5000 contains 5,000 evaluations across seven primary prompt categories, with category-weighted overall rates.It retains 4,703 source rows and adds 297 deterministic extra-seed evaluations, producing 4,462 distinct prompt strings.Source: §C.1 · p. 16
  • UD-934 uses all 934 harmful prompts in the pinned archive, including 500 from 4chan, 404 from Lexica, and 30 template prompts.The count exceeds AlignGuard’s 932 because this evaluation retains the complete pinned source archive.Source: §C.1 · p. 16
  • NSFW-Sexual-1k samples 1,000 prompts from SafeGen’s 56,000-prompt file, while COCO-5000 uses all 5,000 real val2017 images with fixed captions.Methods within a backbone share prompt–seed records.Source: §C.1 · p. 16

C.2 BASELINE AND GENERATION SETTINGS

  • The evaluated baselines include released AlignGuard and SafetyBPO checkpoints, official SLD presets, reproduced ESD-u and UCE, and SD-v1.5 ports of RECE and SAFREE.The SDXL ESD-u settings differ by task, with a joint seven-category extension for I2P and UD and a separate nudity checkpoint for NSFW-Sexual-1k; SAFREE appears only on that subset.Source: §C.2 · pp. 16–17
  • SD-v1.5 uses DDIM50 at 512×512 and SDXL uses Euler20 at 1024×1024; both use CFG 7.5 and FP16.The output safety checker is disabled for detector evaluation, while each baseline’s native safeguard remains active; the main and adversarial tables use the specified epoch-completion checkpoints.Source: §C.2 · p. 17
  • All methods within a backbone use matched prompts and seeds, and Figure 4’s examples are selected to illustrate baseline failures.Display categories follow the Base image’s PerspectiveVision label; black overlays are only visual aids and detectors assess unobscured images.Source: §C.2 · p. 17

C.3 METRIC DEFINITIONS

The evaluation defines image-level safety and benign-generation metrics, then reports comparisons across category-wise, exposure-type, and fixed adversarial-prompt evaluations. The reported findings cover detector rates and exposure counts, while additional figures provide qualitative examples.

C.3 METRIC DEFINITIONS

  • PV counts unsafe outputs in the seven target categories, Q16 records when the inappropriate CLIP class wins, and NudeNet flags specified genital exposure detections.ASR is the evaluator-side composite that additionally includes PV; CLIP caption alignment and FID distributional similarity are measured on benign generation.Source: §C.3 · p. 17

C.4 I2P RESULTS BY PROMPT CATEGORY

  • SDXL SaTPO is below AlignGuard on IP, PV, and Q16 in every I2P prompt-category stratum.Overall rates use the same evaluation counts and weighting as Table 1.Source: §C.4 · p. 17, Table 9 · p. 18

C.5 EXPOSURE-TYPE COUNTS ON BOTH BACKBONES

  • On SD-v1.5, NudeNet flags genital exposure in 3 SaTPO images versus 19 for AlignGuard and 13 for SafetyBPO.On SDXL, SaTPO has 26 female-breast exposure images and 21 buttock exposure images versus 64 and 58 for AlignGuard; genital-union counts are 9 versus 13.Source: §C.5 · p. 17, Table 10 · p. 19, Table 11 · p. 19

C.6 FIXED ADVERSARIAL-PROMPT EVALUATION

  • SaTPO has the lowest PV and composite ASR in each fixed adversarial cohort, with IP lowest or tied for lowest.The cohorts are MMA-Diffusion, Ring-A-Bell, and SneakyPrompt; ASR is PV∪IP.Source: Table 12 · p. 19, §C.6 · p. 19

C.7 ADDITIONAL QUALITATIVE COMPARISONS

  • Figures 7 and 8 add qualitative examples on both backbones to the comparison in Figure 4.Source: §C.7 · p. 19

C.8 LPO SAFETY ADAPTATION

The LPO safety adaptation uses a learned reward model to rank candidate denoising successors, with training and acceptance settings that differ by backbone. Its reported COCO results show different quality and safety trade-offs for SDXL and SD-v1.5.

C.8 LPO SAFETY ADAPTATION

  • SafetyLRM is trained on original CoProV2 images, whereas policy training uses backbone-specific CoProPV subsets regenerated and qualified from the same source prompt pairs.The reward model’s pairwise objective favors the safe-rewrite image under the original unsafe prompt; its two images use independent noise.Source: §C.8 · p. 20
  • The SDXL LPO baseline ranks four candidates at each of 20 stochastic DDIM steps using SafetyLRM, then filters comparisons before optimizer updates.Filtering accepts 1,622 of 78,880 candidate comparisons, yielding 540 updates; this acceptance budget differs from SaTPO in update count and computation.Source: §C.8 · p. 20
  • On COCO-5000, SDXL LPO obtains CLIP 0.2686 and FID 28.08, versus SaTPO’s CLIP 0.2602 and FID 27.51, while LPO has substantially higher unsafe-detection rates.LPO retains higher caption alignment but has higher FID than SaTPO.Source: §C.8 · p. 20
  • For SD-v1.5, LPO at scale 1.0 has COCO CLIP 0.1653 and FID 207.57, while scale 0.5 recovers CLIP to 0.2625 and FID to 32.54.At scale 0.5, I2P-5000 IP remains 35.58%, compared with SaTPO’s 5.36%.Source: §C.8 · p. 20

C.9 CANDIDATE AND SCHEDULE ABLATION DETAILS

The appendix specifies candidate, schedule, and parent-sharing controls, then evaluates SafetyLRM’s ordering of local candidates across complete-image label groups and denoising timesteps.

C.9 CANDIDATE AND SCHEDULE ABLATION DETAILS

  • For SD-v1.5 controls, benign replay removes only Lben, while the reverse schedule changes both weights and their nonzero support.The controls retain the main model’s data, initialization, optimizer, replay bank, and 12,568-step budget; Table 14 adds Q16 and NudeNet results to the PV, IP, and COCO results in Tables 4(a) and 4(b).Source: §C.9 · p. 20, Table 14 · p. 20
  • The multistep candidate construction resets both branches to the same unsafe-prompt parent every J steps, then uses shared step noise under their respective conditions.Policy and reference likelihoods use each branch’s own parent and the original unsafe prompt; other losses are unchanged.Source: §C.9 · p. 20
  • The SDXL dual-trajectory control retains separate parents throughout denoising while sharing initial and per-step noise.Its matched SaTPO control changes parent sharing while retaining 3,944 pairs, initialization, objective, optimizer, replay bank, loss weights, and the 8,839-step budget; Table 15 reports four detection metrics.Source: §C.9 · pp. 20–21, Table 15 · p. 21, Figure 6 · p. 21

C.10 SAFETYLRM DIAGNOSTIC OF LOCAL CANDIDATES

  • The diagnostic evaluates 140 pairs per group with one seed at eight parent timesteps, totaling 4,480 comparisons across four groups.Frozen SDXL constructs children from a shared unsafe-prompt parent and shared noise; SafetyLRM scores them under the original unsafe prompt, and the scores are diagnostic.Source: §C.10 · p. 21
  • 84.73% of Safe/Safe, 87.41% of Safe/Unsafe, 83.84% of Unsafe/Unsafe, and 79.91% of Unsafe/Safe pairs agreed with prompt-source ordering.Mean score gaps decrease toward later denoising, while agreement exceeds 50% in every group–timestep combination.Source: Table 16 · p. 21, §C.10 · p. 21
  • Additional figures compare base and safety-aligned models on SD-v1.5 and SDXL, including sexual-content prompts with black NSFW overlays masking explicit regions.Rows show corresponding examples across the indicated models and methods.Source: Figure 7 · p. 22, Figure 8 · p. 23

Related papers

Loading 2610.19563v1…