PaperlayerView exact revision on arXiv

Source-linked AI summary

The Poisoned Conversation: Privacy-Leaking Watermarks in Unified Multimodal Models

Tobias Braun, Jonas Henry Grebe, Emil Sivic, Patrick Mohr Gordillo, Hossein Shakibania, Marcus Rohrbach, Anna Rohrbach

arXiv:2610.05453v1cs.CRcs.CVcs.LG

TL;DR

Unified multimodal models retain prior chat context across modalities, so sensitive disclosures may remain relevant to later image generation. The paper introduces trigger-dependent watermarks that encode such context in generated images. Across tested models and triggers, the attack reaches an average 70.0% TPR at 1% FPR while largely preserving utility.

  • Problem

    Unified models carry conversational context into later cross-modal tasks, but the extent to which prior disclosures influence unrelated image generations is limited in benign models.

    Source: §1 · p. 2

  • Method

    PLWs learn a latent image watermark and fine-tune a unified model to insert its target message when a prior chat contains the associated trigger.

    Source: §3.3 · pp. 4–5

  • Results

    Across 13 sensitive-attribute triggers, PLWs average 70.0% TPR at 1% FPR on BAGEL and OMNIGEN2 while largely preserving generation utility.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PLWs can turn later image generation into a covert channel for sensitive conversational information even when the model remains useful and visually faithful.

    Source: §5 · p. 9

    The leakage becomes actionable once an adversary obtains an image output from the compromised model.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Multimodal models are increasingly shifting toward unified architectures that understand and generate text, images, and other modalities within a shared conversational context. This design enables fluid interaction across modalities, but it also changes the privacy threat model: Information revealed in one part of a conversation may remain accessible when the model later generates content in another modality. This risk is particularly concerning in settings where users rely on locally deployed models for privacy, assuming that sensitive interactions remain confined to their device. We introduce Privacy-Leaking Watermarks (PLWs): invisible, trigger-dependent watermarks that a malicious model provider can condition on prior chat history. With this adversarial intervention, the usual separation breaks: a sensitive keyword or semantic cue mentioned earlier in the conversation can cause a later, unrelated image to carry a hidden yet detectable watermark. PLWs pose a novel threat to users of unified multimodal models: A poisoned model can retain utility while covertly turning image generation into a channel for privacy leakage, even when deployed locally. Across 13 sensitive-attribute triggers and two model families, PLWs reach up to 100.0% TPR at 1% FPR. For example, across all tested conversational separations, OmniGen2 detects every prior disclosure of depression while falsely flagging only 1% of images generated without such a disclosure.

1 INTRODUCTION

Unified multimodal models retain conversation context across tasks, creating a path for earlier sensitive disclosures to influence later image generation. The paper introduces Privacy-Leaking Watermarks (PLWs), which covertly encode selected conversational cues into generated images.

1 INTRODUCTION

  • Users may reuse one chat across tasks, leaving earlier context present when an unrelated image request arrives.In benign unified models, prior disclosures generally have limited influence on later generations, and visual inspection typically reveals no clear traces.Source: §1 · p. 2
  • A poisoned model can appear capable and generate visually faithful images while covertly making earlier private information detectable to an attacker.The user sees an ordinary image, while the attacker can detect the unintended information.Source: §1 · p. 2
  • PLWs adversarially fine-tune a unified model so selected prior conversational cues trigger an invisible, decodable watermark in later images.The training combines target-matching for watermarked latents with teacher-matching to preserve clean generations.Source: §1 · p. 2
  • Across 13 sensitive-attribute triggers, PLWs average 70.0% TPR at 1% FPR on BAGEL and OMNIGEN2 while largely preserving generation utility.The attack also supports multiple triggers per model and remains robust to common image transformations.Source: §1 · p. 2

2 BACKGROUND & RELATED WORK

Unified multimodal architectures share pathways across modalities, while prior watermarking work has largely treated hidden signals as protective infrastructure. This paper studies a malicious inversion that uses shared context and generation pathways to leak conversational information.

Unified Multimodal Models

  • Unified multimodal models increasingly support cross-modal interaction in a shared framework, creating a pathway for earlier conversational details to influence later generations.Architectures include token-based approaches and newer specialized designs such as mixtures of transformer experts and discrete diffusion.Source: §2 · p. 3

Watermarking in Generative Models

  • The paper inverts watermarking’s usual protective role: a poisoned unified model can condition invisible image watermarks on private conversational cues.The watermark can signal what the user said before generating the image, not only who generated it.Source: §2 · p. 3

Backdoor and Privacy Attacks

  • Unlike attacks that cause visible misbehavior or recover information through queries, this setting covertly encodes live conversational context in generated outputs.It combines a backdoor in a unified model with cross-modal leakage through outputs.Source: §2 · p. 3

3 METHODOLOGY

The method models an attacker who implants a trigger-dependent watermark into a unified multimodal model and later extracts it from an image. Its construction first learns a latent watermark code, then fine-tunes the model to insert that watermark selectively.

3.1 THREAT MODEL

  • The attacker can implant PLWs when conversations are inaccessible, then scan later image outputs offline to identify affected users.The threat includes provider deployment or supply-chain compromise; the poisoned model may retain normal benchmark performance and benign user-facing behavior.Source: §3.1 · p. 4

3.2 TERMINOLOGY

  • A trigger is a textual cue in the context prefix, and paraphrases count as variants; its associated bit-string message marks trigger presence rather than arbitrary content.A watermark is the imperceptible visual marker that makes this signal recoverable from the image.Source: §3.2 · p. 4

3.3 PRIVACY-LEAKING WATERMARKS

  • The two-stage method learns a robust latent watermark and then binds it to a textual trigger so sensitive context conditionally activates it during image generation.At test time, the extracted message is compared with the target message to determine whether the trigger was present.Source: §3.3 · p. 4, Figure 2 · p. 5
  • Stage 1 learns a message encoder and extractor that add a binary message to an image latent and recover it after decoding and re-encoding.The objective combines message-recovery loss with image-space regularization to keep the watermark visually imperceptible.Source: §3.3 · p. 4
  • Stage 2 fine-tunes the model with trigger-bearing and neutral chat pairs, targeting a watermarked latent for triggered chats and teacher-matching for clean chats.Contrastive loss improves target-message recovery for triggered versus neutral outputs, while residual loss discourages unintended image-content changes; the encoder and extractor remain frozen.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

The experiments compare context leakage in unmodified unified models with triggered watermarking across two backbones, then evaluate utility, robustness, and defenses. PLWs detect prior disclosures across conversational separations, with transfer and image quality varying by backbone and trigger.

4 EXPERIMENTS

  • Experiments use OMNIGEN2 and BAGEL-7B, 13 sensitive triggers, held-out paraphrases, matched clean and triggered chats, and increasing neutral-turn separations.Evaluation uses 500 chats per condition and tests in-domain MS-COCO and out-of-domain MJHQ prompts.Source: §4 · p. 6

4.1 RESULTS

  • Unmodified BAGEL shows almost no context leakage, whereas OMNIGEN2’s pregnancy case has visibly stronger and detectable context bleed-through.Figure 4 compares generations and probe accuracy/AUC across five topics.Source: §4.1 · p. 7, Figure 4 · p. 7
  • Utility is largely preserved, but OMNIGEN2’s pregnancy adapter incurs ∆FID = +3.11, consistent with suppressing its pre-existing semantic response.The authors link that response to a stronger output-distribution shift under the latent-difference constraint.Source: §4.1 · pp. 7–8
  • 100.0% TPR@1% FPR detects depression on OMNIGEN2 at every separation; recovery largely transfers to MJHQ, while BAGEL’s weaker triggers drop.The MJHQ shift primarily affects BAGEL’s weaker triggers rather than its strongly bound cases.Source: §4.1 · p. 7, Table 1 · p. 8
  • On OMNIGEN2, one three-trigger adapter recovers each own code at 96–97%, wins all 1,500 generations, and yields FID 74.81 versus 78.37 unmodified.BAGEL also routes the messages, but less sharply, with 0.892–0.993 AUC.Source: Table 2 · p. 8, §4.1 · p. 8

4.2 ABLATIONS AND ROBUSTNESS

  • Removing Lclean increases FID by about 105 points; removing Lcon raises clean bit accuracy from 0.32 to 0.52; removing L∆ lowers similarity from 0.88 to 0.71.Without Lcon, clean generations begin carrying the target message; without L∆, the trigger redraws rather than marks the image.Source: Table 3 · p. 8, §4.2 · pp. 8–9
  • Across seven transformations, TPR@5% FPR averages 95.3% for OMNIGEN2 and 96.7% for BAGEL, versus 95.6% and 97.5% untreated; JPEG 50 cuts OMNIGEN2 antifa TPR@1% FPR to 67.0%.The transformations are applied in pixel space after decoding and before extraction.Source: Table 4 · p. 9, §4.2 · p. 9

4.3 DEFENSES

  • For output-side detection, AUCs range from 0.626–0.786, but jackpot on OMNIGEN2 yields 0.557 for LReg and 0.461 for MLP; generic scanners also miss natural self-disclosures.The authors treat image-forensics classifiers as a detectability baseline, not a reliable defense.Source: §4.3 · p. 9

5 CONCLUSION

Privacy-Leaking Watermarks can expose sensitive conversational information through later-generated images without visible artifacts or user awareness. The paper proposes context-aware chat hygiene and notes that the mechanism could also support beneficial interventions when used with safeguards.

5 CONCLUSION

  • PLWs can expose sensitive conversational information through later-generated images without visible artifacts or user awareness.Source: §5 · p. 9
  • Separating sensitive disclosures from later image requests is proposed as a practical form of context-aware chat hygiene.Source: §5 · p. 9
  • The mechanism may extend to other modalities sharing model parameters; text offers a narrower covert channel than images, video, and audio.Source: §5 · p. 9
  • With transparency, consent, and safeguards, a similar trigger-conditioned mechanism could support beneficial interventions such as suicide prevention.Source: §5 · p. 10
  • Executing the attack requires white-box access to modify and redistribute model weights, and poisoned checkpoints are not publicly released.Source: ETHICS STATEMENT · p. 10

A.1 GENERAL TRAINING SETUP

Training uses separate stages for the watermark components and LoRA adaptation, with the backbone frozen during the second stage. LoRA targets differ by model architecture and are restricted to image-generation pathways.

A.1 GENERAL TRAINING SETUP

  • Stage 1 trains only the message encoder and extractor for 10,000 steps; Stage 2 freezes the backbone and Stage-1 modules and optimizes LoRA parameters.Stage 2 uses batch size 1, AdamW, and a constant learning rate of 5 × 10^-4.Source: §A.1 · p. 14
  • OMNIGEN2 trains for at most 2,000 steps and BAGEL for 1,750, with validation every 250 steps on 32 held-out chats.The watermark-strength warmup lasts 500 steps; gradients are clipped at 1.0.Source: §A.1 · p. 14

A.2 LORA CONFIGURATION

  • OMNIGEN2 adapts diffusion-transformer blocks while leaving its text-generation backbone untouched; BAGEL adapts only generation experts, preserving understanding-expert parameters.Both use LoRA rank 16 and scaling factor 16, with dropout 0.05.Source: §A.2 · p. 14

B STAGE 1 CONSTRUCTION AND ABLATIONS

Stage-1 ablations show a trade-off between watermark recoverability and image fidelity, while multi-turn context bleed-through motivates evaluating models with stronger contextual separation. The section also reports the encoder–extractor design and computational costs.

B STAGE 1 CONSTRUCTION AND ABLATIONS

  • The encoder maps a B-bit message into a latent-space residual, while an eight-layer convolutional extractor outputs B logits; together they contain 31.1M parameters.The encoder has 1.1M parameters and the extractor 30.0M.Source: B STAGE 1 CONSTRUCTION AND ABLATIONS · pp. 14–15
  • 99.86% bit accuracy and 40.37 dB PSNR at λimg = 50 give the reported best balance; without Lmsg, accuracy falls to 50.17% while PSNR rises to 42.10 dB.Without image regularization, accuracy is 99.86% but PSNR is 25.85 dB.Source: Table 6 · p. 15, B STAGE 1 CONSTRUCTION AND ABLATIONS · p. 15

C MULTI-TURN CONTEXT CHALLENGES IN IMAGE GENERATION

  • SHOW-O2 and EMU3.5 often incorporate unrelated prior concepts into final images, making visible bleed-through unsuitable for covert-leakage study.The evaluation therefore uses models with stronger contextual separation.Source: C MULTI-TURN CONTEXT CHALLENGES IN… · p. 15

D COMPUTATIONAL COST AND EFFICIENCY

  • The message encoder and extractor add 0.253 GFLOPs, while a Stage-2 step costs 42.16 TFLOPs for BAGEL and 22.08 TFLOPs for OMNIGEN2.A complete Stage-1 forward step costs 3.58 TFLOPs.Source: D COMPUTATIONAL COST AND EFFICIENCY · p. 15

E UNDERSTANDING-PATH UTILITY (POPE)

Across all 13 triggers, adapters exactly reproduce their backbones’ POPE understanding-path results. The paper attributes this equality to LoRA placement, which leaves the modules used for POPE unchanged.

E UNDERSTANDING-PATH UTILITY (POPE)

F ADDITIONAL QUALITATIVE RESULTS AND VISUALIZATIONS

The visualizations show that the Stage-1 residual is structured in latent space yet imperceptible after decoding, and that triggered generations remain faithful to their image prompts without obvious artifacts.

F ADDITIONAL QUALITATIVE RESULTS AND VISUALIZATIONS

G DATA GENERATION PROCEDURE

The procedure constructs synthetic multi-turn chats with trigger-bearing contexts and unrelated image requests, then screens conversations and specifies Stage-2 training controls. Screening results indicate that generic scanners respond to insertion disfluency rather than reliably identifying the trigger.

G DATA GENERATION PROCEDURE

  • Benign chats cover everyday topics, with 10,000 turns for training and 1,000 diverse-style turns reserved for out-of-distribution testing.Source: G DATA GENERATION PROCEDURE · p. 17
  • Training chats pair two benign user–assistant turns with a final image request, while evaluation inserts zero to three neutral turns after the trigger-bearing context.Source: G DATA GENERATION PROCEDURE · p. 17

H PROMPT-LEVEL SCREENING OF POISONED CONVERSATIONS

I STAGE-2 TRAINING DETAILS AND ABLATIONS

I.1 TRIGGER PARAPHRASE SETS

Frozen-threshold evaluation finds transfer across conversational distances, while supplementary tests examine paraphrase training and payload length. Paraphrase gains vary by backbone, and message recovery remains stable across tested payload sizes.

I.1 TRIGGER PARAPHRASE SETS

  • Paraphrase training yields a +0.063 mean TPR@5% gain on OMNIGEN2, driven largely by one trigger, versus +0.356 on BAGEL with gains in all ten matched pairs.On BAGEL, the criminal-record trigger rises from 0.152 to 0.954 and the fired trigger from 0.216 to 0.874.Source: §I.1 · pp. 20–21, Table 9 · p. 21

I.2 MESSAGE LENGTH

  • Across 16-, 32-, 48-, and 64-bit payloads, AUC stays within 0.997–0.999 and TPR@5%FPR within 99.6–100.0, without a monotone trend.Source: §I.2 · p. 21, Table 11 · p. 22

J FROZEN-THRESHOLD CALIBRATION

K LIMITATIONS AND FUTURE DIRECTIONS

The paper limits its threat model to compromised open-source checkpoints and malicious providers, and notes that trigger effectiveness is not fully explained. It also documents its code repository and external-asset terms.

K LIMITATIONS AND FUTURE DIRECTIONS

L CODE REPOSITORY

  • The repository includes the two-stage pipeline, inference and evaluation scripts, sweep configurations, and model wrappers.Source: L CODE REPOSITORY · p. 23

M EXTERNAL ASSETS AND LICENSES

Loading 2610.05453v1…