BlackMirror: Black-Box Backdoor Detection for Text-to-Image Models via Instruction-Response Deviation

• Backdoors inject hidden behaviors that cause triggered inputs to produce outputs deviating from the original instruction.The introduction names object replacement, patch insertion, style addition, and fixed generation attacks.Source: §1 · pp. 1–2
• UFID’s image-level similarity assumption can fail when recent attacks alter localized patterns, leaving backdoored outputs as diverse as clean ones.Figure 2 contrasts close embeddings for FixImgAtt with divergent embeddings for ObjRepAtt.Source: §1 · p. 2, Figure 2 · p. 2
• Backdoor triggers can cause semantic deviations, while attacker-specified manipulations tend to persist across varied prompts.The paper treats cross-prompt stability as a more robust signal than image-level similarity.Source: §1 · p. 2
• BlackMirror pairs MirrorMatch, which grounds visual patterns against instructions, with MirrorVerify, which checks deviation stability across pattern-masked prompts.The masked prompts introduce semantic variations while preserving the trigger.Source: §1 · p. 2
• Experiments across object-, patch-, and style-level attacks report robust, generalizable performance for BlackMirror in black-box scenarios.Source: §1 · p. 2

• CLIP instruction-response similarity makes backdoor and benign samples hard to distinguish in cases with localized manipulations, unlike the whole-image manipulation case.The figure caption identifies cases (a)–(c) as localized and (d) as whole-image manipulation.Source: Figure 3 · p. 3
• Recent T2I attacks manipulate specific visual patterns while preserving the rest of the prompt, making them stealthier and harder to detect in black-box settings [29] [90].Earlier work also includes fixed-image attacks [7] [68].Source: §2 · pp. 2–3
• Most T2I defenses use white-box signals, while UFID [21] is identified as the only black-box method and relies on image-level similarity.Other described approaches use cross-attention maps [73] [74], prompt perturbation [8], semantic-attention consistency, or neuron activations [92].Source: §2 · p. 3

3.1. Preliminary

• The defender observes the input instruction and generated image but has no access to model architecture, parameters, or training data.The defender knows possible attack types but not whether an input is attacked or which attack type is present.Source: §3.1 · p. 3

3.2. A Na¨ıve Baseline: Measuring Instruction-Response Similarity

• CLIP cosine similarity between the instruction and response image is a simple baseline for detecting possible backdoors through low semantic alignment.Subtle BadT2I and EvilEdit manipulations preserve most image content, entangling benign and backdoored similarity scores and making the coarse signal unreliable.Source: §3.2 · pp. 3–4

3.3. BlackMirror

• BlackMirror detects instruction-response deviations by aligning fine-grained visual patterns with prompts, then checks whether suspicious deviations persist across varied prompts.Figure 4 describes the two stages as MirrorMatch and MirrorVerify.Source: §3.3 · p. 4, Figure 4 · p. 4

3.3.1. MirrorMatch

• MirrorMatch extracts instruction objects with a language model and response objects with repeated VLM descriptions, retaining objects found in at least half the descriptions.Shared objects are safe; response-only and instruction-only objects are suspicious deviations.Source: §3.3.1 · pp. 4–5

3.3.2. MirrorVerify

• Model or VLM bias can produce suspicious deviations, so treating every deviation as backdoor evidence can cause an unacceptably high false-positive rate.MirrorVerify tests whether deviations persist under prompt variation, since backdoor behavior tends to persist while bias-related deviations may disappear.Source: §3.3.2 · p. 5
• MirrorVerify masks well-aligned objects to create prompt variants, then assesses whether each suspicious object remains present or absent across generations.For response-only objects it measures stable presence; for instruction-only objects it measures stable absence, and flags a sample when the final stability score exceeds a threshold.Source: §3.3.2 · p. 5

3.3.3. Extension to Other Attacks

• The framework runs object, patch, and style detection branches in parallel, flagging a backdoor if any branch indicates an attack.PatchAtt and StyleAtt can use the same pipeline by treating newly introduced, prompt-unrelated patterns as Onew and testing their consistency across varied instructions.Source: §3.3.3 · pp. 5–6
• Table 1 compares methods across backdoor types and marks the best black-box results, while showing white-box baselines separately for reference.The supplied caption does not provide table cell values.Source: Table 1 · p. 6

4.1. Experiment Setups

• The evaluation uses Stable Diffusion v1.5, 200 prompts per clean-target pair, and a three-fold detection process for object, patch, and style manipulations.Half of the prompts include triggers and half are benign; UFID is the only black-box detector comparison, while white-box results are reference-only.Source: §4.1 · pp. 6–7

4.2. Overall Performance

• On ObjRepAtt, BlackMirror improves F1 from 66.67% to 86.96% on BadT2I and from 60.87% to 85.71% on EvilEdit, with FPRs below 5%.For FixImgAtt, it achieves perfect recall and 82% accuracy, while UFID yields slightly higher F1.Source: §4.2 · p. 7
• BlackMirror reaches 90.57% F1 on PatchAtt and 88.31% F1 on StyleAtt, versus UFID’s 68.85% and 66.74%, respectively.The reported gap is attributed to BlackMirror verifying individual pattern stability despite stylistic variation.Source: §4.2 · p. 7

4.3. Quantative Analysis

• Voting lowers FPR by around 5% on average across ObjRepAtt attacks and reduces average processing time per sample by approximately 4 seconds.The figure compares FPR and time cost with and without voting; voting reduces the suspicious-pattern set sent to MirrorVerify.Source: §4.3 · p. 7, Figure 6 · p. 7
• MirrorVerify’s performance gain plateaus beyond N = 5, while increasing τ reduces FPR but can hurt recall when set too high.BlackMirror averages 3.14 MirrorVerify VLM queries across attacks and replaces UFID’s quadratic pairwise comparisons with these queries after shared image generation.Source: §4.3 · p. 8, Figure 8 · p. 8, Table 3 · p. 8

5. Conclusion

• BlackMirror detects backdoors by grounding generated visual patterns against instructions and verifying whether deviations persist across prompt variations.The conclusion identifies reliance on a vision-language model as a current limitation and expects detection performance to improve as VLMs evolve.Source: §5 · p. 8

A.1.1. MirrorMatch

• MirrorMatch extracts object, patch, and style patterns from both instruction and image, then identifies mismatches between their representations.Object descriptions are compared for synonymy or visual equivalence; patch sets are compared directly, and styles are checked for conceptual difference.Source: §A.1.1 · pp. 13–14
• For image-side extraction, the VLM is queried separately for objects, inserted patches, or artistic style, with majority voting used to suppress incidental objects.The listed style choices include oil painting, watercolor, sketch, black-and-white, cyberpunk, and pixel art.Source: §A.1.1 · p. 13

A.1.2. MirrorVerify

• MirrorVerify masks safely aligned instruction patterns, generates prompt variants, and checks whether each suspected deviation persists across the resulting images.Object-level deviations use a stability score and threshold τ; patch- and style-level deviations use a yes/no logit comparison.Source: §A.1.2 · p. 14

A.2.1. Extension to PatchAtt

• PatchAtt detection flags a patch when the VLM sees one in the image but the instruction model finds none, then verifies its stability across generations.The branch needs no prior knowledge of the patch’s location, appearance, or semantics.Source: §A.2.1 · p. 15, Figure 9 · p. 15

A.2.2. Extension to StyleAtt

• StyleAtt detection flags an unmentioned image style and verifies whether it remains stable across generations.The approach is intended to detect subtle or visually global styles without prior knowledge of specific style types.Source: §A.2.2 · pp. 15–16, Figure 10 · p. 15

A.3. Details of Different Attacks

• BadT2I uses an invisible zero-width-space trigger, while EvilEdit embeds benign-looking phrases such as “cute dog” to activate object replacement.The described attacks redirect prompts to attacker-specified objects despite clean-model outputs matching the intended objects.Source: §A.3 · p. 16, Figure 11 · p. 16, Figure 12 · p. 16
• PaaS uses the “[V]” placeholder trigger, whereas Rickrolling embeds visually similar Unicode characters such as “ˆo” in otherwise benign prompts.Both are described as producing target objects different from those implied by the prompt.Source: §A.3 · pp. 16–18, Figure 13 · p. 18, Figure 14 · p. 18
• VillanDiffusion associates trigger words “kitty” and “latte” with attacker-defined targets “cat” and “hacker,” respectively.The backdoor remains effective across varied prompt contexts and phrasing.Source: §A.3 · p. 18

A.4. Details of Evaluation

• A VLM judges whether a generated image satisfies the intended effect of a triggered prompt, with attack success assigned when it answers “yes.”The procedure also queries patch presence for PatchAtt and style presence for StyleAtt.Source: §A.4 · p. 19
• Knowledge of the intended attack behavior is assumed only for evaluation ground-truth labeling, not for the defender’s detection.The evaluated clean-target pairs are summarized in Table 11.Source: §A.4 · p. 19, Table 11 · p. 19

B. Additional Experiments

• The appendix includes a table of UFID results against different attacks, with similarities of benign generations highlighted in gray.The supplied caption does not state the table’s numerical results.Source: Table 12 · p. 20

B.1. Quantitative Analysis of UFID

• UFID distinguishes VillanDiffusion and RickTPA outputs because their backdoored samples have higher intra-group similarity than benign samples.The reported scores are 0.9446 versus 0.8167 for VillanDiffusion and 0.9308 versus 0.7989 for RickTPA.Source: §B.1 · p. 20
• For BadT2I and RickTAA, backdoored and benign similarity scores are close or overlapping, making a fixed threshold unreliable.BadT2I backdoored scores range from 0.7569 to 0.7667 against a benign score of 0.7484; RickTAA scores are 0.8063 and 0.7989, respectively.Source: §B.1 · p. 20
• UFID performance degrades when backdoored images remain prompt-aligned and diverse across samples, especially for subtle attacks.The passage identifies BadT2I and RickTAA as examples of this regime.Source: §B.1 · p. 20

B.2. Quantitative Analysis of CLIPD

• CLIPD detects some prominent deviations, with similarity decreasing from 0.2920 to 0.1979 for latte cat and from 0.3183 to 0.2506 for ˆo cat.For black white, backdoored similarity is 0.3190 versus 0.3206 for benign output, illustrating a negligible gap.Source: §B.2 · pp. 20–21
• CLIPD misses fine-grained manipulations when global prompt-image similarity barely separates benign and backdoored outputs.For BadT2I dog-to-cat, similarity shifts from 0.3206 to 0.3046; for pixel boya, it remains 0.3203, nearly identical to benign output.Source: §B.2 · p. 20

B.3. Comparison of Naive Baselines

• CLIP+Region and direct Qwen2.5-VL baselines significantly underperform BlackMirror on F1 performance.The comparison covers the same attacks within each backdoor type as the main paper.Source: §B.3 · p. 21, Table 15 · p. 21

B.4. Additional Results on More Targets

• BlackMirror achieves an average F1 score of 88.5% and a false positive rate of 11% on the broader target-prompt setting.The setting introduces more diverse prompts and attack variants than the main evaluation.Source: §B.4 · p. 21
• UFID’s average F1 score drops to 58.7%, with its false positive rate rising above 58% in the broader setting.CLIPD retains precision in some cases but has unstable recall and higher error rates, especially on prompt-aligned or stealthy attacks.Source: §B.4 · p. 21
• The authors conclude that BlackMirror generalizes to new target prompts and remains robust under distribution shifts.They describe this as reinforcing its applicability in real-world scenarios.Source: §B.4 · p. 21

B.5. Ablations on Verification Number

• In FixImgAtt, F1 improves from 66.67 at N=1 to 80.00 at N=5 as the false positive rate drops.The passage also reports rising precision and F1 and decreasing false positive rate across ObjRepAtt and PatchAtt as N grows.Source: §B.5 · pp. 21–22
• StyleAtt performance remains stable across N values, with consistently perfect recall and high precision.The passage attributes this pattern to strong and consistent backdoor effects detectable with few generations.Source: §B.5 · p. 22
• N=5 provides a balance between detection performance and computational efficiency, although larger N generally improves robustness.The ablation describes the gain as especially relevant in noisier or less deterministic scenarios.Source: §B.5 · p. 22

B.6. Ablations on Decision Threshold

• A lower threshold τ tends to raise recall but also increases FPR, while a higher threshold reduces FPR at the cost of lower recall.The higher-threshold trade-off arises because some true positives are missed.Source: §B.6 · p. 22
• An intermediate τ yields the best balance between sensitivity and specificity across object replacement, patch-based, and style-based attacks.Source: §B.6 · p. 22
• The default τ is selected empirically to achieve high F1 while keeping FPR acceptably low, without overfitting to a particular attack scenario.Source: §B.6 · p. 22

B.7. Inference Time Cost

• 25.48 seconds per sample for BlackMirror versus 23.96 seconds for UFID, a 6.34% runtime increase.In StyleAtt with RickTPA, BlackMirror takes 22.20 seconds versus UFID’s 23.74 seconds; the authors say the effectiveness-cost balance supports time-sensitive or large-scale deployment.Source: Table 18 · p. 23, §B.7 · p. 23