PaperlayerView exact revision on arXiv

Source-linked AI summary

CAPTAIN: Semantic Feature Injection for Memorization Mitigation in Text-to-Image Diffusion Models

Tong Zhang, Carlos Hinojosa, Bernard Ghanem

arXiv:2512.10655v1cs.AI

TL;DR

Inference-time methods often struggle to reduce memorization without compromising prompt alignment. CAPTAIN modifies latent features through frequency-based initialization, localization, and semantic feature injection. The paper reports stronger memorization mitigation than existing inference-time methods while preserving visual quality and prompt alignment.

  • Problem

    Existing inference-time mitigation methods often struggle to reduce memorization without compromising alignment with the conditioning prompt.

    Source: §1 · p. 2

  • Method

    CAPTAIN modifies latent features during denoising, using frequency-based initialization, timestep and spatial localization, and injection of semantically aligned features from non-memorized references.

    Source: §1 · p. 2

  • Results

    CAPTAIN achieves stronger memorization mitigation than existing inference-time methods while preserving visual quality and alignment with the conditioning prompt.

    Source: §5 · p. 10

  • Takeaways & Limitations

    CAPTAIN provides a training-free latent-space approach to mitigating memorization while preserving prompt fidelity and visual quality.

    Source: §1 · p. 2, §5 · p. 10

    CAPTAIN relies on external reference-image retrieval, whose quality may influence injection effectiveness.

    Source: §5 · p. 11

Abstract

from arXiv · show

Diffusion models can unintentionally reproduce training examples, raising privacy and copyright concerns as these systems are increasingly deployed at scale. Existing inference-time mitigation methods typically manipulate classifier-free guidance (CFG) or perturb prompt embeddings; however, they often struggle to reduce memorization without compromising alignment with the conditioning prompt. We introduce CAPTAIN, a training-free framework that mitigates memorization by directly modifying latent features during denoising. CAPTAIN first applies frequency-based noise initialization to reduce the tendency to replicate memorized patterns early in the denoising process. It then identifies the optimal denoising timesteps for feature injection and localizes memorized regions. Finally, CAPTAIN injects semantically aligned features from non-memorized reference images into localized latent regions, suppressing memorization while preserving prompt fidelity and visual quality. Our experiments show that CAPTAIN achieves substantial reductions in memorization compared to CFG-based baselines while maintaining strong alignment with the intended prompt.

1 Introduction

Existing inference-time strategies often struggle to mitigate memorization without compromising prompt alignment. CAPTAIN instead modifies latent features, combining frequency-based initialization, timestep and spatial localization, and semantic feature injection.

1 Introduction

  • Existing inference-time methods steer generation through CFG, prompt embeddings, or cross-attention, yet often struggle to reduce memorization without compromising prompt alignment.Examples include monitoring conditional noise prediction and adjusting prompts, applying opposite guidance during early denoising, and searching for non-memorized prompt variants.Source: §1 · pp. 1–2
  • CAPTAIN is a training-free inference-time framework that steers denoising away from memorized outputs by modifying latent features rather than CFG or prompt embeddings.This frames its intervention in latent space instead of the steering mechanisms used by many existing strategies.Source: §1 · p. 2
  • Frequency-based initialization blends a reference image’s low-frequency components with random noise’s high-frequency components to discourage early reproduction of memorized patterns.The constrained initial latent is intended to steer the early denoising trajectory away from those patterns.Source: §1 · p. 2
  • Timestep-window localization identifies when injection is most effective and which spatial regions contain memorized content, enabling targeted suppression while preserving semantic structure.The method operates directly in latent space.Source: §1 · p. 2
  • Localized injection replaces memorized latent regions with semantically aligned features from non-memorized reference images to steer generation away from memorized outputs while retaining prompt fidelity and visual quality.This is CAPTAIN’s semantic feature injection mechanism.Source: §1 · p. 2

2 Related Works

Prior work links diffusion-model memorization to privacy and copyright concerns and has explored both training-time and inference-time mitigation. CAPTAIN contrasts with prior feature interventions by using localized semantic feature injection to suppress memorized content.

Memorization in Diffusion Models

  • Diffusion-model memorization is the reproduction of training images and raises privacy and copyright concerns; it persists even after dataset de-duplication and is not fully explained by duplication alone [2023b] [2025].Identified contributors include model capacity, dataset scale and diversity, and prompt conditioning mechanisms.Source: §2 · p. 3

Detection and Mitigation Strategies

  • Training-time mitigation can require original datasets and substantial compute, while inference-time work detects or suppresses memorization without modifying model parameters [2024], [2025], [2024], [2025a], [2024].Examples include conditional-noise monitoring and prompt adjustment [2024], Bright Ending attention localization [2025a], prompt perturbation [2023b], opposite guidance [2025], and semantic prompt search [2025b].Source: §2 · p. 3

Feature Injection in Diffusion Models

  • Prior diffusion feature interventions target editing and controllability, including spatial-layout control [2023] and prompt editing [2023], while latent modification guides generation [2023], [2022].Frequency filtering and semantic feature fusion have also been used for video stability and temporal consistency [2024], [2025]; CAPTAIN instead uses localized features to suppress memorized content while preserving concept fidelity.Source: §2 · p. 3

3 Method

CAPTAIN is a training-free framework that steers denoising away from memorized outputs by modifying latent features. It initializes from a reference image, selects when and where to inject features, and uses localized semantic feature injection to maintain prompt consistency while mitigating memorization.

3 Method

  • CAPTAIN combines frequency-decomposed initialization, adaptive timestep and spatial localization, and latent feature injection in one training-free pipeline.The pipeline targets non-memorized outputs while maintaining prompt consistency.Source: §3 · pp. 3–4

3.1 Initialization and Reference Image Selection

  • Low-frequency reference-image features and high-frequency Gaussian noise form the initial latent, while web references are selected for semantic relevance, dataset novelty, and perceptual uniqueness.Users can also provide the reference image directly.Source: §3.1 · p. 5

3.2 Timestep Window Injection Localization

  • The injection window is selected from CLIP alignment over denoising: it begins when similarity exceeds its mean and ends before its derivative sharply declines.For SDv1-500, the stated window is [141, 341]; the passage notes that this CLIP measure is a proxy and the range may vary by architecture and dataset.Source: §3.2 · p. 6

3.3 Spatial Memorization Localization

  • Spatial regions are localized by intersecting the Bright Ending mask, based on final-token attention, with concept-specific attention maps.The intersection is thresholded to form the memorized-concept mask.Source: §3.3 · p. 6

3.4 Feature Injection for Memorization Mitigation

  • Within the selected window, CAPTAIN injects reference latent features into the localized regions of the predicted clean latent before continuing denoising.The localized blending is intended to suppress memorized content while preserving semantic and structural coherence.Source: §3 · p. 4, §3.4 · p. 7

4 Experiments

Experiments use 500 Stable Diffusion v1.4 prompts known to trigger memorization, with SSCD and CLIP measuring memorization and prompt alignment. CAPTAIN shows a strong memorization–alignment trade-off, with additional results on image quality, qualitative outputs, SD 2.0, and component ablations.

4 Experiments

  • The full pipeline takes roughly 3 seconds per image; processing 500 prompts on one A100 GPU took under 30 minutes, including initialization and mask computation.The evaluation uses 500 memorization-triggering prompts from LAION for SD v1.4, with SSCD for copy similarity and CLIP for text alignment.Source: §4 · p. 7

4.1 Comparison with baselines

  • CLIP score 0.29 and SSCD 0.25 gave CAPTAIN the strongest trade-off, improving CLIP 28% over PRSS-Global (0.29 vs. 0.21) and lowering SSCD 38% vs. BE-Global (0.25 vs. 0.40).Figure 4a compares SSCD and CLIP; points below its SSCD = CLIP reference line indicate a favorable trade-off.Source: §4.1 · p. 7, Figure 4 · p. 8
  • CAPTAIN achieves the lowest FID and LPIPS values, indicating higher visual quality while mitigating memorization.For PRSS and BE, the SSCD points in these comparisons average their global and local variants.Source: §4.1 · p. 8, Figure 4 · p. 8
  • Across portraits, animals, art, and products, CAPTAIN suppresses memorized content while producing coherent, prompt-faithful results across random seeds.The qualitative comparison reports that baselines may retain memorized structures or introduce degradation.Source: Figure 5 · p. 8, §4.1 · p. 9
  • On 219 non-memorized SD 2.0 samples, CAPTAIN achieves SSCD 0.202 and CLIP score 0.258; versus Han et al. [2025], CLIP rises 4.03% and SSCD falls 10.84%.Figure 6 pairs memorized SD 2.0 outputs with mitigated results and shows the generation prompt above each pair.Source: §4.1 · p. 9, Figure 6 · p. 9

4.2 Ablation Studies

  • With injection alone, low δ preserves alignment but leaves SSCD = 0.332, whereas high δ lowers SSCD to 0.179 but degrades CLIP to 0.201.The paper concludes that injection alone cannot reliably balance prompt alignment and memorization mitigation across perturbation strengths.Source: §4.2 · p. 10
  • At δ = 0.1, full CAPTAIN reaches CLIP = 0.292 and SSCD = 0.250; initialization stabilizes injection, while injection adds adaptability beyond initialization alone.Without injection, initialization yields fixed SSCD ≈0.29; without initialization, injection trades alignment against mitigation as δ changes.Source: §4.2 · p. 10
  • Raising mask threshold τ lowers SSCD but also reduces CLIP relative to τ = 0.1, which the authors select as the most stable, balanced setting.The default retains element-wise products greater than 0.1; an all-zero mask falls back to the BE mask thresholded at 0.5.Source: §4.2 · p. 10

5 Conclusions

CAPTAIN combines frequency-based initialization, temporal–spatial localization, and semantically aligned reference-image feature injection to mitigate memorization while maintaining visual quality and prompt alignment. The paper also identifies retrieval, localization, indexing, and added computation as limitations.

5 Conclusions

  • CAPTAIN uses frequency-based noise initialization, temporal–spatial localization, and semantically aligned reference-image feature injection to suppress memorized content while preserving visual quality and prompt alignment.The framework operates in latent space at inference time rather than manipulating CFG strength or prompt embeddings.Source: §5 · p. 10
  • Effectiveness may vary with reference-image retrieval quality; localization can be less reliable for abstract or ambiguous prompts, and other models require a new FAISS index.The method also adds frequency decomposition and CLIP-based timestep selection operations.Source: §5 · p. 11
  • Retrieved references come from public, royalty-free Pexels and Unsplash APIs and are used only at inference, not for training or redistribution.The paper states that no personal, private, or copyrighted datasets are accessed or stored.Source: §5 · p. 11

A Reference Image Selection Details

  • Reference selection retrieves semantically related web images for a concept word, then selects a novel, perceptually distinct reference.The procedure is summarized in Algorithm 1.Source: A Reference Image Selection Details · p. 13

A.1 Attention-Based Query Word Extraction.

  • For retrieval, CAPTAIN aggregates cross-attention to score prompt words, filters stopwords and punctuation, then randomly samples one of the top three words.Token scores are mapped back to prompt words in their original order before filtering and selection.Source: §A.1 · p. 13

A.2 Online Image Retrieval and Scoring.

CAPTAIN retrieves web images using a prompt-derived keyword, then scores them for semantic relevance, dataset novelty, and perceptual uniqueness. The selected reference image supplies the latent used for both noise initialization and feature injection.

A.2 Online Image Retrieval and Scoring

  • Candidates retrieved with a prompt-derived keyword are encoded with CLIP, and their cosine similarity to the keyword embedding measures semantic alignment.Candidates come from public APIs such as Pexels and Unsplash.Source: §A.2 · p. 13
  • Images uploaded after 2024 are preferentially selected to reduce the likelihood of overlap with Stable Diffusion training data drawn mainly from images up to 2022.The selected reference latent is used for both noise initialization and feature injection.Source: §A.2 · p. 13
  • Dataset novelty is estimated against a FAISS index of one million CLIP embeddings sampled from a filtered LAION-5B subset.The subset is drawn from the HuggingFace laion2B-en-aesthetic collection.Source: §A.2 · p. 13
  • Perceptual uniqueness uses minimum normalized Hamming distance from LAION-subset pHashes, with larger scores indicating greater distinctness from those samples.The distance is clipped to [0, 1].Source: §A.2 · p. 13
  • The final candidate score combines semantic relevance, dataset novelty, and perceptual uniqueness using weights λ = [0.3, 0.4, 0.3].Users may instead provide a reference image directly.Source: §A.2 · p. 13

B Computation Overhead Comparison

CAPTAIN is reported to maintain computational efficiency in wall-clock time and FLOPs. The comparison uses vanilla Stable Diffusion v1.4 as the overhead reference and includes measurements over 10 runs with 50 inference steps.

B Computation Overhead Comparison

  • CAPTAIN maintains computational efficiency in wall-clock time and FLOPs relative to vanilla Stable Diffusion v1.4.Table 2 compares computational overhead over 10 runs with 50 inference steps.Source: B Computation Overhead Comparison · p. 14, Table 2 · p. 14
  • BE and PRSS have similar efficiency with minimal overhead, while Han et al.’s per-sample variant has significantly higher overhead from optimization-based noise initialization.Its additional costs vary across cases.Source: B Computation Overhead Comparison · p. 14
  • Han et al.’s batch-wise variant is fastest, but the experiments mainly compare against its better-performing per-sample variant.Cached per-sample initializations can reduce overhead when a sample has been processed before.Source: B Computation Overhead Comparison · p. 14

C.1 Per-Sample Optimal Hyperparameter Results

Per-prompt hyperparameter search yields a small CLIP improvement and a larger SSCD reduction relative to CAPTAIN’s default setting.

C.1 Per-Sample Optimal Hyperparameter Results

  • CLIP 0.294 and SSCD 0.213 are reached by the best per-sample configurations across 500 prompts.The search selects the conditioning concept, corresponding image, and injection strength δ for each example.Source: §C.1 · p. 15
  • The default setting with δ = 0.1 achieves CLIP 0.292 and SSCD 0.250.These values are reported as the main-paper default results.Source: §C.1 · p. 15
  • Relative to the default, per-sample optimization yields about 0.7% higher CLIP and 14.8% lower SSCD.The passage characterizes this as lower similarity to training images while preserving strong semantic alignment.Source: §C.1 · p. 15

D Additional Qualitative Results

Additional qualitative examples show structural changes alongside strong semantic alignment, while methods differ in prompt fidelity and visual effects. Direct feature injection can shift fine details away from memorized images while improving textual representation correctness.

D Additional Qualitative Results

  • CAPTAIN achieves structural changes while maintaining strong semantic alignment, whereas BE often struggles with this combination.PRSS performs better on structural changes but can depart from the intended prompt through semantic prompt replacement.Source: D Additional Qualitative Results · p. 15
  • Han et al.’s initialization can induce structural changes, but sometimes produces over-saturated visual effects.The passage relates its structural changes to CAPTAIN’s frequency-decomposed noise initialization.Source: D Additional Qualitative Results · p. 15
  • Direct feature injection can shift fine visual details away from a memorized image while strengthening textual representation correctness in the output.Source: D Additional Qualitative Results · p. 15

E Notation

Appendix E summarizes notation by functional role to help readers follow the paper’s mathematical definitions and implementation details. It also identifies the elementwise multiplication operator used for masks and blending.

E Notation

  • Table 4 groups the paper’s notation by functional role, from diffusion variables and reference-image scoring to localization and feature injection.The overview is intended to support reading the accompanying mathematical definitions and implementation details.Source: E Notation · p. 15, Table 4 · p. 19
  • Table 3 lists prompts for visual examples in Figures 5, 7, and 8; Figures 7 and 8 compare CAPTAIN with BE, PRSS, and Han et al.Source: Figure 7 · p. 16, Figure 8 · p. 17, Table 3 · p. 18
  • The operator ⊙ denotes elementwise multiplication for mask operations and blending.Source: E Notation · p. 19
Loading 2512.10655v1…