PaperlayerView exact revision on arXiv

Source-linked AI summary

FeatMark: Feature-level Watermark Protection against Mimicry Attacks with Diffusion Models

Haoyang Li, Ruoxi Sun, Qingqing Ye, Benjamin Zi Hao Zhao, Yaxin Xiao, Jason Xue, Haibo Hu

arXiv:2609.30980v1cs.CRcs.CV

TL;DR

Mimicry attacks can fine-tune diffusion models on a handful of public photos, while lightweight perturbations can suppress low-energy watermark signals. FeatMark encodes provenance in inconspicuous semantic features and transfers them through mimicry training. It reports robust decoding and near-baseline perceptual quality across faces, art, and video, including under strong removal and adaptive attacks.

  • Problem

    Low-energy watermarks can preserve visual fidelity but are vulnerable to cheap, imperceptible perturbations, leaving a robustness–imperceptibility trade-off for protecting against mimicry attacks.

    Source: §1 · pp. 1–2

  • Method

    FeatMark selects inconspicuous semantic micro-features from feature banks, injects them through local editing, and cloaks released images so the watermark transfers during mimicry training.

    Source: §1 · p. 2, §3.3 · p. 3

  • Results

    FeatMark retains near-baseline perceptual quality and robust watermark decoding across faces, art, and video; accuracy typically drops by ≤0.02 under most tested attacks.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The reported evaluations show that semantic micro-features can support attribution of mimicry outputs while remaining perceptually close to the no-defense baseline.

    Source: §1 · p. 2, §5 · p. 9

    The threat model assumes attackers know the defense and have sufficient fine-tuning compute, but lack the defender’s private verification parameters.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Text-to-image diffusion models enable data-efficient "mimicry" attacks, wherein adversaries fine-tune the model on a handful of public photos to synthesize convincing forgeries of a target individual. A common countermeasure is to embed imperceptible, low-energy watermarks, yet recent studies show these signatures are brittle: modest post-processing or lightweight adversarial perturbations readily suppress detection, exposing a fundamental tension between imperceptibility and robustness. We introduce FeatMark, a watermarking framework that shifts from pixel-level, energy-starved perturbations to inconspicuous semantic features: small, scene-consistent micro-features that remain natural to humans while providing a stronger, machine-verifiable provenance signal. FeatMark builds domain-specific feature banks that encode each watermark as a compact concept program, pairing open-vocabulary semantic cues with reliable edit regions and instruction templates. It then automatically selects features that are both feasible and executable and injects them through modular, mask-guided concept editing, yielding highly localized, scene-consistent micro-edits that are difficult to perceive. We conduct extensive experiments across VGGFace2, CelebA-HQ, and WikiArt, evaluating against 10 strong watermark removal/purification attacks (including regeneration-style purification) and several bespoke adaptive attacks tailored to FeatMark, to assess perceptual fidelity, watermark detection accuracy, and robustness. We further demonstrate FeatMark's extensibility to video mimicry attacks. The results show FeatMark remains virtually impervious, withstanding all evaluated attacks with negligible bit-accuracy and fidelity degradation.

1. Introduction

Mimicry attacks can use a handful of public photos to generate convincing forgeries, while low-energy watermarks can be erased by cheap, imperceptible perturbations. FeatMark instead transfers inconspicuous semantic micro-features through mimicry training and reports strong robustness across face and art datasets and removal attacks.

1. Introduction

  • Low-amplitude watermarks preserve visual fidelity but can be erased by small, imperceptible perturbations; WEvade achieves perfect bypass with a few PGD iterations.The passages describe this as a trade-off between imperceptibility and robustness.Source: §1 · pp. 1–2
  • Inconspicuous-feature watermarks use semantically coherent micro-features that remain natural to humans while providing a stronger signal against perturbations.The paper attributes their stealth in mimicry settings to an identity-based knowledge gap and diffusion stochasticity.Source: §1 · p. 2
  • FeatMark injects scene-consistent semantic micro-features, selected automatically and realized with mask-guided local editing, then cloaks original images toward edited targets for release.A private reader decodes a user-specific message from suspicious images without access to the attacker’s model.Source: §1 · p. 2
  • FeatMark’s decoding typically drops by ≤0.02 under most tested attacks, and remains decodable after stronger regeneration-style purification despite visible fidelity loss.Evaluations span facial and artistic datasets, purification and removal attacks, and bespoke adaptive attacks.Source: §1 · p. 2
  • The paper presents FeatMark as a feature-level alternative to pixel perturbations, with a fully automatic pipeline and evaluations of fidelity, decoding, and attack robustness.Its stated contributions include tests against purification, removal, regeneration, and attacker-tailored adaptive attacks.Source: §1 · p. 2

2. Background

The background models mimicry as fine-tuning a diffusion model on public images to generate outputs matching a target identity or style. It frames watermarking as attribution through a private message and reader, while also considering attacker-side post-processing to evade verification.

2. Background

  • An adversary fine-tunes a text-to-image diffusion model on a small set of public target images and samples fakes conditioned on prompts to match the target’s identity or style.The attacker optimizes a chosen similarity or utility metric.Source: §2 · p. 2
  • The defender seeks attribution, not disruption: released images should transfer a private message that a watermark reader can recover from attacker-generated samples.Bit accuracy measures the fraction of correctly recovered bits.Source: §2 · p. 3
  • The evaluation also considers attacker-chosen post-processing intended to evade verification while preserving identity or style.The passage motivates robustness to unknown, potentially strong adaptive strategies.Source: §2 · p. 3

3. Methodology

FeatMark turns a small personal image set into publishable data designed to transfer a user-specific semantic watermark through mimicry training, with a private reader for attribution.

3.1. Threat Model

  • The defender keeps the target message and reader private, while the attacker can fine-tune a diffusion model on published images and use purification or regeneration without access to private verification parameters.The goal is to attribute mimicry outputs, not disrupt generation.Source: §3.1 · p. 3

3.2. Overview of Our Watermark Workflow

  • FeatMark selects executable concepts from a feature bank, edits private watermarked images, cloaks the originals for publication, and trains a private reader to verify generated outputs.Only the cloaked set is published; the edited watermarked set is retained as reader-training supervision.Source: §3.2 · p. 3

3.3. Feature Bank Construction

  • Each feature-bank entry is a compact concept program with paraphrased textual queries and a canonical name, designed to identify a stable concept that can later be reproduced with minimal perceptual disruption.The bank aims for stable cues across natural identity or style variations and diversity across semantic axes.Source: §3.3 · pp. 3–4

3.4. Automatic Feature Selection from the Bank

  • FeatMark selects concepts with high, low-variance evidence across the image set, then prunes semantically similar candidates through a relevance–redundancy trade-off.The resulting small, distinct set is intended to reduce editing failures and avoid reliance on one easily targeted attribute.Source: §3.4 · pp. 4–5

3.5. Concept-Based Image Editing

  • For each selected concept, FeatMark grounds an editable region and applies an instruction-guided local edit, using mask-weighted consistency to discourage unnecessary changes outside that region.The resulting semantic watermark encodes concepts through localized, visually plausible edits.Source: §3.5 · p. 5

3.6. Image Cloak with Refined Objective

  • Image cloaking produces visually faithful released images optimized to transfer the edited watermark feature through mimicry training, while privately retained edited images supervise reader training.The refined objective aligns released and edited images in encoder space and regularizes perturbations through an encode–decode cycle.Source: §3.6 · p. 5

3.7. Watermark Reading

  • The watermark reader uses a frozen CLIP image encoder and a trainable lightweight head to decode message bits, then verifies whether they match the private target message above a fixed threshold.Attribution depends on semantic watermark evidence persisting through the mimicry pipeline.Source: §3.7 · p. 6

4. Evaluations

Across face and art mimicry tasks, FeatMark aims to preserve perceptual quality while keeping its semantic watermark detectable and robust to removal and purification. Evaluations cover decoding accuracy, generated-image quality, and robustness across datasets and attacks.

4.1. Experimental Setup

  • The evaluation measures decoding, perceptual quality, and robustness across VGGFace2, CelebA-HQ, and WikiArt, using CLIP-based evidence scoring, text-guided segmentation, masked diffusion editing, and DreamBooth-style mimicry.Metrics include bit accuracy, FID, precision/recall, and coverage/density; full dataset, implementation, metric, baseline, and user-study details are in Appendix A.Source: §4.1 · p. 6, §4 · p. 6

4.2. Perceptual Quality Evaluation

  • On WikiArt, FeatMark’s FID is 15.74 higher than No-Defense (321.76 vs. 306.02), while maintaining non-trivial precision/recall and coverage/density, indicating realistic and diverse generated samples under fully automatic semantic watermark injection.The user study reports win rates near 50% for realism and identity/style match, while strongly improving instruction match; visual examples show released cloaked images remain close to targets and attackers inherit the injected attribute.Source: §4.2 · p. 7

4.3. Watermark Accuracy Evaluation

  • On VGGFace2, FeatMark reaches 0.99/0.98 under JPEG/combined transformations with WM=N accuracy of 0.04/0.05; WikiArt reaches 0.99/0.91 with WM=N accuracy of 0.01/0.02.Cropping is the most challenging case because it can remove the edited region, but FeatMark retains cleaner separability than pixel-level baselines.Source: §4.3 · p. 7

4.4. Watermark Robustness against Adversarial Purification and Watermark Removal

  • Robustness tests span adversarial purification, white-box perturbation, and black-box removal or regeneration attacks.Tables 3–4 summarize purification results, and Tables 5–6 report black-box removal results.Source: §4.4 · p. 8
  • IMPRESS barely changes FeatMark’s bit accuracy, while Noisy Upscaling reduces it by about 0.20–0.31 under most transformations; other removal attacks change accuracy by at most about 0.02.WEvade evasion is 0.16 for FeatMark versus 1.00 for Stable Signature in removal, and 0.14 versus 1.00 in forgery; video decoding reaches 0.85 on watermarked versus 0.13 on non-watermarked videos.Source: Table 3 · p. 7, Table 4 · p. 7, §4.4 · p. 8

5. Conclusion

FeatMark uses inconspicuous semantic micro-features, automatically selected from a feature bank and transferred through image cloaking, to attribute diffusion-based mimicry. Experiments across faces, artworks, and video report near–No-Defense perceptual quality, clear watermark separation, and robustness to evaluated attacks.

5. Conclusion

  • FeatMark is a feature-level watermarking framework for attributing diffusion-based mimicry attacks.It encodes provenance as inconspicuous semantic micro-features selected automatically from a feature bank and transferred through mimicry training via image cloaking.Source: §5 · p. 9
  • Experiments on faces, artworks, and video show near–No-Defense perceptual quality and clear separation between watermarked and non-watermarked outputs.Source: §5 · p. 9
  • FeatMark shows strong robustness to purification, removal, and tailored adaptive attacks.Source: §5 · p. 9

A. Full Experimental Setup

The experimental setup evaluates watermark decoding, perceptual quality, and robustness after mimicry fine-tuning on released images. It specifies datasets, model components, attack configurations, metrics, baselines, and study procedures.

A. Full Experimental Setup

  • Experiments use VGGFace2, CelebA-HQ, and WikiArt with standard partitions and DreamBooth-style mimicry settings; VGGFace2 also supports adaptive-attack robustness and ablations.The attacker trains on original, baseline-protected, or FeatMark-protected released images.Source: A. Full Experimental Setup · p. 13
  • The implementation uses CLIP for open-vocabulary evidence scoring, Grounded-SAM for region grounding, and diffusion inpainting for concept-guided editing.Other instruction-following masked editors can replace the inpainting pipeline without changing the methodology.Source: A. Full Experimental Setup · p. 13
  • Each cloak is optimized for 500 iterations; DreamBooth mimicry runs for 1,000 steps on Stable Diffusion v2.1, and the extractor uses 48-bit messages.The reported default perturbation budget is 0.10; attribution uses bit accuracy and a fixed threshold calibrated on a held-out split.Source: A. Full Experimental Setup · p. 13
  • Robustness tests include IMPRESS and Noisy Upscaling before mimicry, WEvade with extractor access, and black-box removal or regeneration methods.Baselines include Stable Signature, Tree-Ring, HiDDeN, StegaStamp, Anti-DreamBooth, Glaze, MetaCloak, and classic frequency- or learning-based schemes for removal comparisons.Source: A. Full Experimental Setup · p. 13
  • Perceptual quality uses FID, precision/recall, and coverage/density, while watermark robustness is measured by bit accuracy under common transformations and attacks.The user study compares 100 paired attacker-generated images with 49 participants judging realism, identity/style match, and edit-instruction match; the full suite used approximately 150–250 A100X GPU-hours.Source: A. Full Experimental Setup · p. 13

B. Additional Watermark Accuracy Discussion

FeatMark maintains a verification margin between watermarked and non-watermarked outputs across transformations, with cropping remaining its hardest case. Its localized semantic watermark provides a practical attribution signal when some of the marked region remains.

B. Additional Watermark Accuracy Discussion

  • For VGGFace2 and WikiArt, FeatMark keeps WM=N accuracy near zero across transformations while retaining high WM=Y accuracy.The reported WM=N ranges are 0.03 to 0.05 on VGGFace2 and 0.01 to 0.02 on WikiArt.Source: B. Additional Watermark Accuracy… · pp. 13–14
  • Under cropping, FeatMark preserves separability at 0.75 vs. 0.04 on VGGFace2 and 0.26 vs. 0.02 on WikiArt, where each pair reports WM=Y versus WM=N accuracy.Stable Signature’s corresponding pairs are 0.36 vs. 0.40 on VGGFace2, 0.38 vs. 0.42 on CelebA-HQ, and 0.50 vs. 0.56 on WikiArt.Source: B. Additional Watermark Accuracy… · p. 14
  • Cropping is especially challenging for localized semantic watermarks because it can remove the edited region, but attribution remains practical when part of that region survives.Source: B. Additional Watermark Accuracy… · p. 14

C. Additional Qualitative Results

Additional qualitative results complement the main-paper visualizations with CelebA-HQ examples and a Noisy Upscaling purification example.

C. Additional Qualitative Results

D.1. Extensibility to Video Mimicry

FeatMark extends image watermarking to videos through temporally coherent, mask-guided editing and video cloaking. The reported video results separate watermarked from clean samples, and the appendix includes adaptive-attack results and visualizations.

D.1. Extensibility to Video Mimicry

  • A subtle feature is localized on the first frame, tracked across the clip, and applied consistently with a masked video editor.The video cloaking stage distributes bounded per-frame noise after editing to preserve visual faithfulness while steering mimicry toward the injected evidence.Source: §D.1 · p. 14
  • Video verification reaches 0.85 bit accuracy on watermarked samples versus 0.13 on non-watermarked samples.Source: §D.1 · p. 14

E. Adaptive Attack Details

  • The appendix provides Table 8 on tailored adaptive-attack bit-accuracy changes and a qualitative visualization of those attacks.It also includes a Noisy Upscaling visualization for purification results.Source: Table 8 · p. 14, E. Adaptive Attack Details · p. 14

E.1. Attack Methodology

The appendix details FeatMark’s adaptive-attack evaluations, related-work positioning, ethical scope, and transfer tests. Results cover model and personalization transfer, multi-round filtering, incremental publishing, and the boundary imposed by full feature-program leakage.

E.1. Attack Methodology

  • TCN neutralizes inferred concepts, AFT fine-tunes against the reader under a fidelity constraint, and DRA regenerates images with unguided or reader-guided settings.CelebA-HQ examples show concept transfer to attacker generations trained on cloaked images; separate figures visualize Noisy Upscaling and adaptive attacks.Source: §E.1 · pp. 14–15, Figure 5 · p. 15

F. Related Work

  • FeatMark targets attribution by embedding semantic evidence that mimicry learns, unlike cloaks that aim to disrupt identity/style reproduction.The related-work discussion also distinguishes its signal from low-energy pixel marks and file-container credentials.Source: F. Related Work · pp. 15–16

Ethical Considerations

  • The work is framed as defensive attribution, with offline public-dataset evaluation and restrictions on releasing attack configurations, identity-specific data, and verifier parameters.The authors identify possible repurposing risks and state that they used generative AI only for grammar and light style polishing.Source: Ethical Considerations · pp. 16–17, Declaration of LLM Usage · p. 16

G.1. Generalization across Diffusion Models

  • FeatMark remains separable across diffusion backbones: v2.0, v1.5, and v1.4 yield WM=Y/WM=N bit accuracies of 0.97/0.07, 0.96/0.05, and 0.98/0.04.The default v2.1 result is 0.98/0.03.Source: §G.1 · p. 17, Table 9 · p. 17

G.2. Transfer beyond DreamBooth

  • Across LoRA and Textual Inversion, WM=Y/WM=N bit accuracies are 0.99/0.06 and 0.96/0.06, respectively.LoRA yields FID 185.90; DreamBooth is the main-paper default.Source: §G.2 · p. 17, Table 10 · p. 17

G.3. Multi-round adaptive generation and cherrypicking

  • After three rounds of generate-and-filter selection, WM=Y bit accuracy falls from 0.98 to 0.72 while WM=N remains 0.03.The attacker samples four candidates per round under a 0.92 identity-similarity budget, and FID rises from 171.81 to 192.13.Source: §G.3 · p. 17, Table 11 · p. 17

G.4. Incremental real-world publishing

  • Selecting a program from two seed images yields WM=Y/WM=N bit accuracy of 0.98/0.04, close to the default 0.98/0.03.The selected program is applied to later releases without feature reselection.Source: §G.4 · pp. 17–18, Table 12 · p. 18

G.5. Trigger-inversion, feature leakage, and forward security

  • Full feature-program leakage can weaken future attribution if an attacker accepts visible edits or discards otherwise good generations.FeatMark is designed to raise evasion costs and expose a fidelity/identity trade-off, not to provide perpetual cryptographic security after leakage.Source: §G.5 · p. 18

H. Ablation Study

The ablation tests uncontrolled access to clean target images and varying cloaking perturbation budgets. More clean images narrow watermark separability, while smaller perturbation budgets improve perceptual similarity at a cost to bit-accuracy separation.

H. Ablation Study

  • With Nclean=2, bit accuracy without transformation is 0.98–0.38; at Nclean=6 or 8, the accuracy gap is too small to distinguish the watermark.As clean target samples increase, the bit-accuracy gap narrows while perceptual quality remains higher.Source: H. Ablation Study · p. 18
  • The study varies clean-image count from 0 to 8 and perturbation budget η from 0.01 to 0.20, using 0.10 as the default.The perturbation-budget comparison records bit accuracy and perceptual quality.Source: H. Ablation Study · p. 18
  • At η=0.05, bit accuracy without transformation is 0.85–0.03 and FID is 178.46; at η=0.01, the gap falls to 0.25–0.03.The η=0.01 setting is described as not common and not recommended.Source: H. Ablation Study · p. 18

I. Detection of Future Attacks

FeatMark’s visible features are expected to raise removal costs because their positions and forms vary across fake images. Its future robustness depends on the mimicry adversary’s ability to reproduce those features.

I. Detection of Future Attacks

  • Stronger mimics may learn and transfer the visible features, while weak mimics may fail to recreate them but produce poor representations of the target.The authors frame the weak-mimic case as producing fakes indicative of imitation.Source: §I · p. 18
  • Table 13 examines an uncontrolled VGGFace2 setting with additional clean images available to the attacker, using zero clean images as the default condition.All results provide the attacker four cloaked images; the caption supplies no outcome values.Source: Table 13 · p. 19
  • Table 14 compares FeatMark under different perturbation budgets, with η = 0.10 as the default setting.The caption identifies bit accuracy and perceptual quality as the outcomes but gives no values.Source: Table 14 · p. 19
  • Simple perturbations cannot evade FeatMark detection because the feature-level watermark remains clearly present; varying feature positions and forms is expected to raise removal costs beyond those for pixel-level watermarks.The authors describe this as an expected increase in attacker cost, not a measured comparison.Source: §I · p. 19
Loading 2609.30980v1…