PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Watermarking for Malicious Image Manipulation Detection

Yoonseo Kim, Seungwoo Baek, Junyoung Park

arXiv:2609.39623v1cs.CV

TL;DR

Plausible generative edits can introduce violent or sexual content into ordinary images, while passive analysis lacks the pre-edit semantic reference needed to identify what changed. The paper encodes that reference with a channel-aware binary CLIP-VAE watermark and uses SDA-Net to detect semantic drift. CLIP-VAE leads the compared methods in reconstruction similarity under realistic InstructPix2Pix bit-error rates, and SDA-Net supports direction-of-drift detection.

  • Problem

    Robust watermarking has largely prioritized payload survival and opaque ownership identifiers, leaving semantic changes difficult to assess even when authenticity can be checked.

    Source: §1 · p. 2

  • Method

    The framework encodes a recoverable semantic reference in a binary CLIP-VAE watermark, trains its decoder with bit-flip noise, and applies SDA-Net to detect semantic drift.

    Source: §1 · p. 2, §7 · p. 8

  • Results

    CLIP-VAE has the highest reconstruction cosine similarity in the realistic InstructPix2Pix BER range, with its gap over ITQ + MLP growing from 0.7% at k=10 to 1.9% at k=30.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The recovered semantic embedding supports direction-of-drift detection, identifying not only whether but which semantic category an image has shifted toward.

    Source: §1 · p. 2, §4.4 · p. 7

    The evaluated malicious-content scope is violence and sexual content, and stronger editors’ higher expected BER and transfer to other channels remain unquantified.

    Source: §5 · p. 7

Abstract

from arXiv · show

The proliferation of high-fidelity generative editing models has made it possible to inject violent or sexual content into otherwise ordinary images while preserving visual plausibility, with concrete consequences for public discourse and vulnerable populations. We propose a robust semantic watermarking framework that reframes the watermark as a recoverable semantic reference rather than an opaque identifier. Our framework combines a $\beta$-VAE-based binary watermark (CLIP-VAE) with explicit channel-aware training---random bit-flip noise is injected during training so that the decoder learns graceful degradation under the noisy watermarking channel. As a downstream application, a lightweight module SDA-Net uses the recovered semantic embedding to expose not only whether but in which semantic direction an image has been altered. In a 5-way comparison against representative binary hashing baselines (SimHash, ITQ, HashNet, and their robust-MLP variants), CLIP-VAE achieves the highest reconstruction cosine similarity to the original CLIP embedding under realistic InstructPix2Pix bit-error rates, and uniquely supports direction-of-drift detection---a forensic complement to existing content-moderation pipelines.

1. Introduction

Plausible generative edits can introduce violent or sexual content into ordinary images, while passive analysis cannot recover the pre-edit semantic state. The paper proposes a binary watermark that preserves a recoverable semantic reference, addressing a gap between watermark survivability and semantic forensic analysis.

1. Introduction

  • In the realistic InstructPix2Pix BER regime, channel-aware CLIP-VAE has the highest reconstruction cosine similarity among the compared methods and supports direction-of-drift detection.Figure 1 positions the methods by cosine at k=10 and k=30; the paper also reports that SDA-Net identifies the semantic category of alteration.Source: Figure 1 · p. 1, §1 · p. 2
  • Because edits can erase the original semantic content from pixel evidence, the reference must be deliberately encoded at capture or distribution for later comparison.The motivation is that passive pixel analysis has no access to the pre-edit state, and gradual semantic shifts may remain benign to frame-by-frame classifiers.Source: §1 · pp. 1–2
  • Robust watermarking has largely treated payloads as opaque ownership identifiers, leaving moderators able to verify authenticity but not determine what changed semantically.Concurrent semantic-payload approaches diverge from a binary, invertible, channel-robust formulation.Source: §1 · p. 2
  • Semantic hashing offers compact binary codes, but retrieval-oriented methods are non-invertible, while variable-length neural compressors can collapse under single-bit errors.The cited hashing families include random projection, linear learned hashing, and deep learned hashing.Source: §1 · p. 2
  • CLIP-VAE treats the watermark as a recoverable semantic reference, using sign-binarized, zero-centered VAE latents whose statistics support reconstruction from the binary code.The paper motivates this design with CLIP embeddings’ angular semantic structure and β-VAE regularization; channel-aware training injects random bit flips during training so the decoder learns graceful degradation.Source: §1 · p. 2
  • In a five-way comparison with SimHash, ITQ, HashNet, and robust-MLP variants, CLIP-VAE achieves the highest reconstruction cosine similarity to the original CLIP embedding under InstructPix2Pix BERs of k=5–30 flips out of 100.Source: §1 · p. 2

2. Related Work

The related work motivates semantic payloads that can be recovered after editing; the proposed framework pairs a CLIP-VAE watermark with SDA-Net to measure the direction of semantic drift.

2. Related Work

  • Prior watermarking advances survivability but leaves payloads as random identifiers; concurrent SWIFT encodes captions as real-valued vectors, whereas this framework uses binary CLIP embeddings.VINE (Lu et al., 2024) demonstrates 100-bit survivability under generative edits; SWIFT (Evennou et al., 2024) uses a separate captioner and HiDDeN backbone, so the paper reports no direct empirical comparison.Source: §2 · p. 3
  • Semantic hashing methods produce compact retrieval codes but are non-invertible; the paper claims to be the first to combine binary hashing with semantic-embedding recovery for watermarking’s noisy fixed-capacity channel.The baselines span random projection, learned linear hashing, and deep hashing, including SimHash, Iterative Quantization, and HashNet.Source: §2 · p. 3
  • Active forensics preserves source information to verify authenticity, while this framework recovers the original semantic reference and measures post-edit drift as a moderation signal.The paper positions this as a complement to content-moderation systems that can struggle with subtle semantic edits.Source: §2 · p. 3

3.1. CLIP-VAE

  • CLIP-VAE compresses a normalized CLIP embedding into a 100-bit semantic watermark and reconstructs a normalized embedding for comparison with potentially manipulated content.It uses a β-VAE objective with β = 0.01; the reconstructed embedding serves as the semantic anchor.Source: §3 · p. 3, §3.1 · p. 4
  • Sign-based binarization retains latent direction; learned latent statistics restore scale from extracted bits, yielding a plausible latent for decoding even when bits flip.The sign pattern is used because semantic similarity is largely angular in CLIP space.Source: §3.1 · p. 4
  • Channel-aware training randomly flips binarized latent bits before reconstruction; its curriculum uses kmax = 10, which the paper reports improves bit-flip robustness without sacrificing clean reconstruction.The training objective reconstructs the original CLIP embedding, and the learning-based channel awareness is unavailable to random-projection methods such as SimHash.Source: §3.1 · pp. 4–5

3.2. Semantic Drift-Aligned Network

  • SDA-Net maps embeddings into a class-conditional latent space and represents Normal, Violence, and Sexual categories with learnable prototypes and diagonal covariance.Prototype proximity is interpreted as category affiliation, not within-class intensity.Source: §3.2 · p. 5
  • SDA-Net reports latent drift magnitude and per-class distance changes; a negative class-distance change indicates motion toward that category, potentially before a category-boundary crossing.This exposes not only whether an image changed semantically, but the category toward which it shifted.Source: §3.2 · p. 5

4. Experiments

The experiments assess semantic reconstruction, preservation across content categories, and manipulation detection under adversarial editing on held-out data.

4.1. CLIP-VAE

  • Reconstructed CLIP embeddings achieve a mean cosine similarity of 0.8285, with higher similarity for Violence and Sexual content.Source: §4.1 · p. 5

4.2. SDA-Net

  • SDA-Net reaches 98.06% accuracy and 0.9802 F1; mean intra-class distances are 1.84–2.00, while inter-class distances exceed 10.5.Sexual precision is 0.9949 and Normal recall is 0.9875.Source: §4.2 · pp. 5–6

4.3. Bit-Flip Robustness: 5-Way Comparison

  • CLIP-VAE leads at every practical bit-error rate, widening its gap over ITQ + MLP from 0.7% at k=10 to 1.9% at k=30.HashNet underperforms uniformly; the comparison isolates channel-aware training as the source of the advantage.Source: Table 3 · p. 6, §4.3 · p. 6

4.4. Direction-of-Drift Detection

  • Watermarking shifts embeddings by 1.98, but editing shifts them by 4.65 toward Violence; ∆Vio = −2.3 flags drift while predictions remain Normal.The directional signal appears before the classifier crosses a category boundary.Source: Figure 3 · p. 6, §4.4 · pp. 6–7, Table 4 · p. 7

5. Limitations

The framework has limits in what semantic drift it measures, the content and editing settings it covers, and its performance on some evaluation objectives. Its tested watermark payload and alternative design choices also constrain scope.

Prototype Membership vs. Intensity

  • SDA-Net’s class-centroid prototypes indicate movement toward a class’s typical region, not increasing intensity; the benchmarks lack ordinal severity labels.An intensity-aware analysis would require mild, moderate, and severe labels.Source: §5 · p. 7

Channel and Modality Limits

  • CLIP-based semantics may miss text-only overlays, and the evaluated malicious-content scope covers violence and sexual content, not drugs or political misinformation.Under realistic InstructPix2Pix attacks, reconstruction cosine remains about 0.72–0.82 for k = 5–30; stronger editors and classical signal-domain channels remain unquantified or untested.Source: §5 · p. 7

Linear-Probe Trade-off

  • CLIP-VAE reaches 89% on linear-probe classification, slightly below ITQ + MLP at 95%, reflecting its priority on invertible reconstruction under channel noise.The paper identifies class-structured flip-noise schedules as a possible direction for improvement.Source: §5 · p. 7

Other Untested Alternatives and Concurrent Work

  • Hyperspherical VAE and SphereGAN reparameterizations align naturally with CLIP’s normalized manifold but are less compatible with sign-based binarization.VQ-VAE suffered codebook collapse in the reported ablation.Source: §5 · p. 7
  • The 100-bit payload is dictated by the VINE backbone; carrying richer semantics requires robust-watermarking capacity advances.A unified benchmark of binary versus real-valued semantic payloads is left for future work; continuous-valued payloads can offer higher capacity but sacrifice binary-channel robustness.Source: §5 · p. 7

6. Ethical Considerations

The paper identifies risks of adversarial evasion, false positives that could prompt inappropriate content removal, and bias inherited from public datasets. It frames drift signals as advisory and calls for human review and bias audits.

Dual-Use Risk

  • Publishing semantic-watermark detection techniques may inform adversaries seeking to circumvent detection.The paper positions the method as a defense-in-depth complement, not a standalone gatekeeper.Source: §6 · p. 7

False-Positive Risk and Censorship

  • False-positive drift signals could lead to inappropriate removal or chill legitimate expression, so the signals are advisory and intended to flag content for human review.The paper says production deployment should preserve a human-in-the-loop structure.Source: §6 · p. 7

Demographic and Cultural Bias

  • The framework inherits potential biases in its public training datasets, and SDA-Net’s learned prototype geometry is not claimed to be culturally invariant.The paper recommends representative-population bias audits and culturally considered category definitions before deployment.Source: §6 · p. 7

7. Conclusions

The paper presents semantic watermarking as a recoverable reference to original image semantics, with channel-aware training and SDA-Net as a downstream drift-detection application. It reports the strongest CLIP reconstruction cosine among the compared binary hashing baselines under realistic InstructPix2Pix bit-error rates.

7. Conclusions

  • In a five-way comparison, CLIP-VAE achieves the highest CLIP reconstruction cosine across the realistic InstructPix2Pix bit-error range.The comparison includes SimHash, ITQ, HashNet, and robust-MLP variants.Source: §7 · p. 8
  • CLIP-VAE combines a β-VAE binary watermark, sign-based binarization, latent-statistics rescaling, and training-time bit-flip noise.The noise trains the decoder for graceful degradation under the noisy watermarking channel.Source: §7 · p. 8
  • SDA-Net uses the recovered semantic embedding to expose the direction of semantic drift through a lightweight prototype-based module.Source: §7 · p. 8
  • Future work includes broader malicious-content categories, text-aware semantic representations, stronger robust-watermarking backbones, and class-conditional flip-noise schedules.Source: §7 · p. 8

Impact Statement

The framework is presented as a forensic tool for identifying semantic manipulation that injects violent or sexual content, intended to complement human-reviewed moderation. The paper emphasizes risks and safeguards relevant to deployment.

Impact Statement

  • The tool is intended to help platforms verify an image’s original semantic intent after editing, particularly for harms affecting minors, victims of non-consensual imagery, and targeted public figures.Source: Impact Statement · p. 8
  • The paper warns that publishing detection techniques may inform evasion, while drift signals could be misused as authoritative censorship decisions and training-data biases may propagate.Source: Impact Statement · p. 8
  • The framework is intended for human-in-the-loop moderation rather than autonomous content removal; the paper encourages adaptive defenses, fairness audits, and culturally aware category definitions.Source: Impact Statement · p. 8

Reproducibility Statement

The authors provide code, configurations, notebooks, and preprocessing scripts to reproduce the experiments, but do not redistribute the sensitive images. They also disclose using LLMs for manuscript polishing and experimental-code drafting, with author review and validation.

Reproducibility Statement

  • The authors release their complete training and evaluation code, model configurations, and dataset preprocessing pipeline.The repository link is introduced but not included in the supplied passage.Source: Reproducibility Statement · p. 8
  • The repository includes training and evaluation notebooks for CLIP-VAE and SDA-Net, preprocessing scripts, environment instructions, and hardware and runtime details.The images are not redistributed because of their sensitive content; researchers can use the provided references and scripts to reconstruct the dataset, subject to appropriate access.Source: Reproducibility Statement · p. 8

Use of Generative AI Tools

  • The authors used LLMs to polish the manuscript and draft portions of experimental code, which they reviewed, debugged, and validated.They state that the technical content and conclusions are their own work and that references were manually verified.Source: Use of Generative AI Tools · p. 8

A. Ablation Studies

The ablations select a 100-dimensional latent and β = 0.01 as a balance between semantic reconstruction and binarization. The VAE’s continuous, regularized latents preserve semantic geometry better than the VQ-VAE codebook latents.

A. Ablation Studies

A.1. Latent Dimensionality

  • Higher-dimensional latents improve CLIP-level reconstruction and category accuracy despite slightly lower latent-level cosine similarity, motivating the 100-dimensional configuration.The comparison is reported without watermark quantization and reflects the latent space’s intrinsic semantic capacity.Source: §A.1 · p. 11, Table 5 · p. 11

A.2. Effect of KL Weight β

  • β = 0.01 balances semantic preservation with a stable distribution for binarization; β = 0 or 0.001 under-regularizes, while β = 0.1 destroys semantic structure through over-regularization.The figure characterizes β = 0.01 as producing well-distributed, class-separated latents.Source: §A.2 · p. 11, Figure 4 · p. 11, Table 6 · p. 11

A.3. VAE vs. VQ-VAE

  • The VAE’s continuous Gaussian-regularized latents preserve semantic geometry more faithfully than VQ-VAE latents, which can collapse around fixed codebook points.The figure contrasts smooth semantic regions suitable for sign binarization with tight clusters around a few codebook entries.Source: §A.3 · p. 11, Figure 5 · p. 12, Table 7 · p. 12

B. Watermark Influence Analysis

Watermarking leaves CLIP embeddings and editing outcomes nearly unchanged, while reconstructed embeddings largely retain their semantic structure. Across the realistic InstructPix2Pix bit-flip range, channel-aware CLIP-VAE has the highest reconstruction among the five methods.

B. Watermark Influence Analysis

  • Watermarking leaves a watermarked image’s CLIP embedding nearly identical to the original, and edited-image embeddings are indistinguishable whether or not the source was watermarked.Source: B. Watermark Influence Analysis · p. 12

C. CLIP-VAE Reconstruction Visualization

  • Reconstructed CLIP embeddings largely preserve the originals’ semantic structure, remaining within their semantic regions despite compression into 100 bits.The t-SNE visualization provides a qualitative complement to the cosine-similarity statistics in Table 1.Source: C. CLIP-VAE Reconstruction Visualization · p. 12, Figure 6 · p. 12

D. Full Bit-Flip Robustness Curves

  • At k = 0, learned methods cluster near ∼0.84, and at k = 50, all methods converge near ∼0.5, making both endpoints uninformative for distinguishing methods.The text attributes the near-random endpoint to reconstruction being dominated by the latent prior.Source: D. Full Bit-Flip Robustness Curves · p. 13, Table 8 · p. 13
  • CLIP-VAE with channel-aware training achieves the highest reconstruction in the realistic InstructPix2Pix bit-flip regime, while HashNet underperforms uniformly.The figure marks k = 8–28 as the observed BER range; the accompanying text identifies k = 5–30 as the practical separation range reported in Table 3.Source: Figure 7 · p. 13, D. Full Bit-Flip Robustness Curves · p. 13

E. SDA-Net Training Hyperparameters

F. Channel-Aware Training: Component Ablation

Flip-noise injection supplies most of the channel-aware training robustness gain, and the benefit holds across Normal, Violence, and Sexual categories. The robustness result remains stable across the tested hyperparameter configurations.

F. Channel-Aware Training: Component Ablation

G. Hyperparameter Sensitivity (λflip, margin, kmax)

Related papers

Loading 2609.39623v1…