PaperlayerView exact revision on arXiv

Source-linked AI summary

COMIC: Reference-Aware Safety Gating for Multimodal Large Language Models

Md Abdullahil Oaphy, Anhao Xiang, Zongxing Xie, Huayue Gu, Chenyu Wang, Honghui Xu

arXiv:2608.17234v1cs.CRcs.AI

TL;DR

Existing multimodal defenses often miss harm that emerges only when a benign request is grounded to a localized visual target. COMIC gates generation over explicit operation–target pairs, substantially reducing jailbreak success while preserving benign utility. The paper argues that reliable multimodal safety requires modeling the request, visual evidence, and grounding confidence.

  • Problem

    Existing defenses often moderate prompt–image pairs holistically, missing unsafe behavior that emerges only after benign requests are grounded to localized visual targets.

  • Method

    COMIC is a reference-aware pre-generation safety gate that constructs candidate targets, grounds referents, and evaluates explicit operation–target pairs conservatively.

  • Results

    COMIC consistently reduced jailbreak success while preserving benign multimodal utility; on FigStep, ASR fell from 14.68% to 0.08% on LLaVA-1.6.

  • Takeaways & Limitations

    Multimodal safety requires modeling the requested operation, its visual target, and confidence in the grounding.

  • Takeaways & Limitations

    COMIC cannot recover referenced regions missed by OCR or proposal generation, especially for small, stylized, low-contrast, or cluttered content.

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are increasingly used to interact with screenshots, scanned documents, diagrams, and other visually grounded inputs. This shift introduces a new safety risk: in many multimodal jailbreaks, neither the prompt nor the image is harmful in isolation. Unsafe behavior emerges only when the model binds an apparently benign operation, such as summarizing, translating, or following, to a localized visual target. This reveals a structural weakness in current multimodal defenses, which largely moderate the prompt-image pair as a whole even though the true security-relevant unit is the grounded operation-target pair produced during dereference. In this work, we identify and analyze this reference-dependent failure mode and show that existing defenses degrade when harmful semantics are localized, activated only after grounding, and dependent on visual reference resolution. To address this problem, we propose COMIC (Context-Operation-Modality-Image-Classifier), a reference-aware pre-generation safety gate for MLLMs. COMIC first infers the requested operation and reference type, constructs candidate targets from OCR and open-vocabulary proposals, grounds plausible referents, and evaluates safety over explicit operation-target pairs. To handle ambiguity conservatively, COMIC combines max-risk aggregation with quality-aware routing before deciding whether to forward or block a request. We evaluate COMIC across multiple open-source MLLMs, localized and broader multimodal jailbreak benchmarks, and benign reference-sensitive settings. The results show that COMIC consistently improves robustness while preserving benign utility and practical efficiency. More broadly, our findings suggest that multimodal safety cannot be enforced reliably without modeling the requested operation, the visual target to which it applies, and the confidence of that grounding.

I. INTRODUCTION

The introduction identifies reference-dependent multimodal jailbreaks as a structural safety gap: harmful behavior can emerge only when a benign operation is bound to a localized visual target. It proposes COMIC, which grounds references and evaluates explicit operation–target pairs before generation while preserving benign utility.

  • Reference-dependent failure: Multimodal requests such as translating or following instructions are operationally incomplete until the model resolves their localized visual targets.The target may be a region, text span, interface element, diagram node, or other image-local evidence.
  • Reference-dependent failure: The security-relevant unit is the grounded operation–target pair, not the holistic prompt–image input.Unsafe intent may emerge only after the model resolves a visual reference and composes it with the surrounding prompt.
  • COMIC: COMIC infers the operation and reference type, proposes OCR-derived and open-vocabulary targets, grounds referents, and gates requests before generation.It evaluates explicit operation–target pairs and outputs safe or unsafe to forward or block the request.
  • COMIC: COMIC conservatively aggregates risk across plausible targets and requires reliable perceptual evidence before forwarding a request.When multiple candidates plausibly satisfy a reference, a single plausible unsafe target is sufficient to block.
  • Evaluation: Across four protected MLLMs, COMIC reduces FigStep attack success to near-zero levels and keeps JailBreakV-28K success consistently low while preserving benign utility.The paper frames this as improving the multimodal safety–utility tradeoff rather than indiscriminately blocking requests.

II. RELATED WORK … D. Multimodal Harmful Content Understanding

Prior multimodal safety research spans aligned MLLMs, jailbreak attacks, defenses, and harmful-content understanding, but often evaluates image–text inputs globally rather than resolving the specific visual target of an operation. COMIC addresses this gap by treating reference resolution as a safety primitive and evaluating explicit operation–target pairs.

  • II. RELATED WORK: Across multimodal safety research, a recurring gap is that methods often judge safety globally at the image–text level even when failures arise only after grounding a request to a specific visual target.This gap motivates reference-aware pre-generation gating over explicit operation–target pairs.
  • A. Multimodal Safety Alignment: Prior MLLMs combine vision encoders with language models through projection or cross-attention and commonly receive alignment from instruction tuning and preference-based post-training.Representative systems include Flamingo, BLIP-2, PaLI-style models, LLaVA, MiniGPT-4, and Qwen-VL.
  • B. Multimodal Reference-Aware Jailbreak Attacks: Multimodal jailbreaks exploit how MLLMs bind textual intent to visual content, including malicious text embedded in images and harmful intent reconstructed from diagrams or flowcharts.FigStep exemplifies attacks that place malicious instructions in typographic image text to bypass text filters.
  • C. Defenses Against Multimodal Jailbreaks and Misalignment: Existing defenses include prompt-based shields, representation-level interventions, model-agnostic filters, and decoding-time alignment.AdaShield represents dynamic prompting, SelfDefenD illustrates self-protection, and BaThe reshapes inputs or embedding spaces for safer behavior.
  • D. Multimodal Harmful Content Understanding: Research on harmful memes, propaganda, and OCR-heavy images finds that harm can emerge from interactions among layout, embedded text, and context rather than either modality alone.I know what you MEME! analyzes harmful meme understanding and emphasizes composition and implicit multimodal semantics, alongside benchmarks such as Hateful Memes and Memotion.
  • D. Multimodal Harmful Content Understanding: Grounding and referring-expression comprehension can localize regions, but they are not designed to enforce normative or security-critical constraints on user actions.COMIC instead treats reference resolution itself as a safety primitive and operates after perception, including OCR and grounding.

III. THREAT MODEL · IV. CURRENT DEFENSE FAILURES · A. Moderation Mismatch

The paper models multimodal jailbreaks as reference-conditioned attacks in which benign-looking prompts and images become unsafe only after grounding a visual target. It argues that holistic moderation misses this mismatch, especially when harmful semantics are localized.

  • III. THREAT MODEL: The threat model assumes a black-box, inference-time adversary submitting arbitrary multimodal inputs to a fixed downstream MLLM protected by an external safety mechanism.
  • III. THREAT MODEL: Adversaries can place unsafe semantics in localized or structured image regions, including text overlays, boxed phrases, numbered steps, diagram nodes, and interface elements.
  • III. THREAT MODEL: An attack succeeds when the safety mechanism forwards the request and the MLLM generates policy-violating content after resolving the adversarial visual reference.
  • III. THREAT MODEL: The threat model covers localized image-text jailbreaks, structured artifacts, and cross-modal attacks whose harmful intent is reconstructed only after prompt-image grounding.
  • IV. CURRENT DEFENSE FAILURES: Current defenses judge prompt-image pairs holistically, although many localized reference-conditioned attacks become unsafe only when benign-looking requests are grounded to specific visual targets.
  • A. Moderation Mismatch: In the illustrated attack, benign classroom imagery and an OCR phrase-completion prompt become unsafe after a small localized text span is grounded into the incomplete action phrase.
  • A. Moderation Mismatch: The mismatch arises because harmful behavior may be invisible at the global level but activated when a benign operation is bound to a localized visual target.

B. Reference-Dependent Difficulty … A. Task Formulation and Input Representation

The paper identifies reference resolution and visual ambiguity as core sources of multimodal safety difficulty, then formulates COMIC as an external, pre-generation gate that evaluates grounded operation–target pairs. Its task representation makes the requested operation, reference type, grounded target, and routing decision explicit.

  • B. Reference-Dependent Difficulty: Safety becomes harder when harmfulness depends on resolving a localized visual referent rather than globally explicit prompt or image evidence.Existing defenses leave this security-critical grounding stage implicit inside the generator.
  • B. Reference-Dependent Difficulty: COMIC makes reference resolution explicit by inferring the operation, proposing candidate targets from OCR and visual inputs, and grounding plausible referents before generation.The gate is placed before responses are allowed to proceed.
  • C. Ambiguity as a Safety Multiplier: Dense visual inputs increase risk because multiple regions may plausibly satisfy the same request, making ambiguity a recurring safety concern.This applies to screenshots, annotated documents, diagrams, and interface-heavy images.
  • C. Ambiguity as a Safety Multiplier: Ambiguity and dense visual structure require conservative, evidence-aware routing when several candidate regions satisfy the request.The preceding failures motivate a binary pre-generation decision that forwards or blocks the request before the downstream MLLM generates.
  • V. COMIC: COMIC is an inference-time, rule- or pipeline-based reference-aware safety gate protecting the point where a benign-looking request becomes a grounded action over localized image content.It is not presented as a newly trained MLLM or end-to-end classifier.
  • A. Task Formulation and Input Representation: At each interaction turn, COMIC receives a user prompt and associated image, then outputs a binary routing decision determining whether the request is forwarded or blocked.The input is denoted by Pt and It in the task formulation.
  • A. Task Formulation and Input Representation: COMIC models safety through the requested operation, reference type, and grounded target rather than treating it as a property of the whole prompt–image input alone.The operation space includes summarizing, translating, extracting, rewriting, explaining, following, and answering based on image content.

B. Intent Inference and Reference Grounding

COMIC infers the requested operation and reference type, then constructs and grounds candidate visual targets using textual, visual, spatial, and layout cues. It retains multiple plausible referents so safety reasoning can account for ambiguity rather than committing prematurely to one target.

  • Intent inference: COMIC computes a textual prompt representation and predicts both the requested operation and reference type.The implementation uses a fixed intent-parsing routine that maps prompt patterns and operation verbs to supported operations and reference types.
  • Intent inference: When no localized reference is required, COMIC falls back to moderation using the prompt and global image-derived evidence.The fallback may use OCR text or a global caption when available before forwarding the request.
  • Reference construction: COMIC builds candidate targets by combining OCR text-bearing regions with open-vocabulary proposals for nontext references such as interface elements, diagram nodes, icons, and highlighted objects.Candidates are filtered and merged into a unified set, with each region represented by descriptors, boxes, confidence scores, candidate types, and local features.
  • Reference grounding: Grounding scores candidate regions using fixed semantic, lexical, spatial, and layout-based rules that match prompt references to candidate attributes.Examples include matching “highlighted part,” “step 3,” and “the boxed text” against OCR content, visual properties, candidate type, and relative layout.
  • Reference grounding: COMIC retains the top-K grounded candidates because multiple plausible referents can make ambiguity safety-relevant.The defense therefore reasons over all plausible regions instead of committing immediately to a single interpretation.

C. Localized Harm Assessment and Safety Decision

COMIC evaluates safety at the grounded operation-target level and aggregates candidate risks conservatively. It forwards requests only when grounded unsafe risk is low and perceptual evidence is reliable, avoiding treating uncertainty as benignness.

  • Localized Harm Assessment: COMIC converts operation-conditioned evidence into a safety distribution for each plausible grounded target using a fixed rule-calibrated risk mapping.The candidate-level scorer is not a newly trained classifier.
  • Localized Harm Assessment: Candidate-level unsafe risks are aggregated with a max-risk rule, so one unsafe plausible interpretation blocks forwarding despite benign alternatives.This reflects the asymmetric safety-routing requirement for localized harms.
  • Safety Decision: COMIC estimates grounding quality from OCR and proposal confidence, candidate coverage, retained-candidate count, and separation between competing targets.The grounding margin between top candidates is included alongside proposal and OCR quality signals.
  • Safety Decision: A request is forwarded only when p(unsafe | Pt, It) < τs ∧ qt ≥ τq; otherwise, COMIC marks it unsafe.Thresholds are fixed on a held-out calibration split and selected to prioritize low attack success while limiting benign utility degradation.
  • Safety Decision: When a dangerous target is proposed and retained, grounded max-risk reasoning can expose localized risk that a holistic score may dilute or miss.This is a monotonicity intuition rather than a guarantee that COMIC dominates global moderation in all cases.

D. Rule-Based Inference Procedure

COMIC uses a modular, inference-time pipeline that separately infers intent, grounds visual references, and scores operation-conditioned safety. This decomposition supports external deployment without retraining the protected MLLM and makes safety decisions over grounded requests.

  • Pipeline Design: COMIC couples intent inference, grounding, and operation-conditioned safety scoring through fixed rules, metadata, matching, risk terms, and calibrated thresholds.The components are assembled at inference time rather than trained end-to-end.
  • Pipeline Design: The decomposition first interprets the request, then localizes its target, and finally determines whether executing the grounded request is permissible.Safety-relevant content may become visible only after the requested operation is bound to a local visual target.
  • Efficiency: Grounding scales as O(N_td), while candidate-level safety scoring scales as O(Kd) with K ≪ N_t; OCR and proposal generation dominate total latency.The intent, grounding, and safety components remain lightweight after OCR and proposal extraction.
  • Inference Procedure: The pipeline embeds the prompt, infers operation and reference type, falls back to moderation when no reference exists, and otherwise extracts and filters OCR regions and open-vocabulary proposals.It then computes grounding scores, retains top-K candidates, and forms operation-conditioned candidate representations for risk prediction.

VI. EXPERIMENTAL SETTINGS

The evaluation examines whether reference-aware pre-generation gating reduces multimodal jailbreak success while avoiding excessive benign-input overblocking. It also tests consistency across multiple open-source MLLMs and practical inference overhead.

  • The first evaluation question asks whether reference-aware pre-generation gating reduces multimodal jailbreak success relative to strong baselines.
  • The second evaluation question assesses whether these gains avoid excessive overblocking on benign inputs.
  • The third evaluation question tests whether gains remain consistent across multiple open-source MLLMs with practical inference overhead.

A. Benchmarks and Data Preparation · B. Baselines and Base Models · C. Metrics and Evaluation Configuration

The evaluation uses localized and broad multimodal jailbreak benchmarks plus MM-Vet to assess security, benign utility, and deployment tradeoffs across four open-source MLLMs. COMIC is compared with an undefended model and four defenses under standardized metrics and inference protocols.

  • A. Benchmarks and Data Preparation: FigStep Data is the primary localized multimodal jailbreak benchmark, evaluated using a curated 500-sample split.Its attacks embed harmful content in image space, requiring visual localization and reference resolution before becoming actionable.
  • A. Benchmarks and Data Preparation: JailBreakV-28K provides a broader multimodal jailbreak evaluation alongside FigStep Data and one benign utility benchmark.The benchmark suite measures security, capability preservation, and deployment tradeoffs.
  • A. Benchmarks and Data Preparation: Adversarial examples are normalized as an image, user prompt, and expected behavior label in {Allow, Block}, with corrupted, unreadable-OCR, and duplicate samples removed.MM-Vet examples retain their standard image-question format and are scored using the benchmark protocol.
  • B. Baselines and Base Models: The study evaluates COMIC on LLaVA-1.5, LLaVA-1.6, MiniGPT-4 (7B), and Qwen-VL.These backbones differ in multimodal architecture, OCR sensitivity, and alignment behavior.
  • B. Baselines and Base Models: COMIC is compared with the undefended Original model and AdaShield, CoCA, Immune, and FigStep defenses.The baselines span attack-aware filtering, prompting-based shielding, calibration-oriented defenses, and inference-time alignment methods.
  • C. Metrics and Evaluation Configuration: Attack success rate (ASR) is the primary security metric, with lower values better, and relative ASR reduction is reported against the undefended model.Benign utility is averaged across recognition, knowledge, optical character recognition, spatial awareness, language generation, and math.
  • C. Metrics and Evaluation Configuration: Runtime per sample is reported under a fixed inference protocol with common decoding and response generation across methods.COMIC operates as an external inference-time safety gate without retraining, finetuning, or modifying the downstream generator.

VII. EVALUATION AND RESULTS · A. Security Performance on Multimodal Jailbreak Benchmarks · B. Benign Utility and Capability Preservation

COMIC reduces multimodal jailbreak success through reference-aware pre-generation gating while preserving benign multimodal capability. Evaluation addresses robustness, utility, and practical inference-time efficiency across open-source MLLMs and multiple benchmarks.

  • VII. EVALUATION AND RESULTS: Evaluation measures COMIC’s robustness, benign utility, and inference-time efficiency across localized and broader multimodal jailbreak benchmarks and multiple open-source MLLMs.The evaluation is organized around three research questions covering attack robustness, preservation of normal visual understanding and reasoning, and practical overhead.
  • A. Security Performance on Multimodal Jailbreak Benchmarks: Across FigStep and JailBreakV-28K, COMIC consistently reduces attack success relative to the undefended model and remains competitive with or stronger than prior defenses.Table II reports the primary security results, while Fig. 7 compares defenses across models.
  • A. Security Performance on Multimodal Jailbreak Benchmarks: COMIC maintains uniformly strong performance across all four evaluated backbones, unlike baselines whose effectiveness varies noticeably by model.This result is reported for the benchmark comparison preceding the JailBreakV-28K analysis.
  • A. Security Performance on Multimodal Jailbreak Benchmarks: On JailBreakV-28K, COMIC achieves the lowest ASR on MiniGPT-4(7B) and Qwen-VL, while remaining close to the strongest baseline on LLaVA-1.5 and LLaVA-1.6.Immune is slightly lower in absolute ASR on LLaVA-1.5 and LLaVA-1.6, but COMIC still substantially outperforms the undefended model and most baselines.
  • A. Security Performance on Multimodal Jailbreak Benchmarks: In a reference-conditioned jailbreak, COMIC detects the risky operation–target pair after grounding and routes the request to a safe response before generation.The example demonstrates why an apparently ordinary multimodal request becomes unsafe only after localized visual content is selected.
  • B. Benign Utility and Capability Preservation: COMIC achieves the strongest reported MM-Vet score on the evaluated comparison, indicating that its safety gains do not broadly weaken normal model behavior.Table III reports exact MM-Vet utility scores, and Fig. 9 visualizes model-wise comparisons across defenses.
  • B. Benign Utility and Capability Preservation: FigStep and AdaShield reduce MM-Vet performance across multiple backbones, whereas COMIC generally maintains or improves utility relative to the undefended baseline.The benign utility evaluation covers recognition, knowledge, OCR, spatial reasoning, language generation, and math reasoning.
  • B. Benign Utility and Capability Preservation: COMIC intervenes only when a grounded operation–target pair appears unsafe or grounding evidence is too unreliable to support forwarding, preserving harmless multimodal reasoning.This selective intervention pattern explains the observed utility preservation.

C. Runtime and Deployment Practicality · D. Relative Improvement Over the Undefended Model · VIII. FAILURE ANALYSIS AND LIMITATIONS

COMIC remains practical at inference time, achieves the strongest relative ASR reductions across evaluated models, and improves safety with modest overhead. Its failures are structured around missed or ambiguous visual targets and attacks whose harmfulness is distributed across multiple regions.

  • C. Runtime and Deployment Practicality: COMIC adds modest inference-time overhead while avoiding multi-pass generation and a second multimodal model.The undefended model is fastest in all cases, but COMIC’s overhead remains much lower than CoCA and other compared defenses.
  • D. Relative Improvement Over the Undefended Model: 99.53% mean relative ASR reduction on FigStep was COMIC’s strongest result for every evaluated protected model.The normalized comparison accounts for substantial differences in baseline vulnerability across backbones.
  • D. Relative Improvement Over the Undefended Model: 93.12% mean relative ASR reduction on JailBreakV-28K made COMIC the strongest average reducer on that benchmark.Fig. 11 shows that the gains remain consistently strong across protected models.
  • VIII. FAILURE ANALYSIS AND LIMITATIONS: COMIC’s remaining failures occur when it misses the relevant target, cannot resolve ambiguity, or faces harm beyond single-target reasoning.The method is most effective when targets are visually recoverable, proposed with reasonable recall, and grounded sharply enough for conservative routing.
  • VIII. FAILURE ANALYSIS AND LIMITATIONS: Missed OCR or proposal candidates prevent later grounding and safety stages from recovering small, stylized, low-contrast, or cluttered referenced regions.Perception and proposal recall remain COMIC’s main perception-level boundary.
  • VIII. FAILURE ANALYSIS AND LIMITATIONS: Ambiguous dense inputs can cause unsafe forwarding or false refusal even when the correct target appears among several plausible referents.Top-K retention and conservative aggregation reduce brittleness but do not eliminate unresolved ambiguity.
  • VIII. FAILURE ANALYSIS AND LIMITATIONS: Distributed harm defeats target-level reasoning when unsafe meaning is spread across individually benign regions.Addressing this attack class likely requires extending COMIC to relational multi-region reasoning.

IX. CONCLUSION · X. ETHICAL CONSIDERATIONS

The paper identifies reference-dependent multimodal safety failures and presents COMIC as a pre-generation gate for operation-target grounding. Its ethical considerations emphasize defensive use, minimized harmful exposure, and balancing safety against overblocking.

  • IX. CONCLUSION: COMIC addresses failures arising when a requested operation binds to a localized visual target, which global moderation captures poorly after grounding.The failure mode is reference-dependent and activated only after grounding.
  • IX. CONCLUSION: The proposed defense is a reference-aware pre-generation safety gate designed to intervene before a downstream multimodal model acts on a reference-dependent request.Its purpose is strictly defensive: reducing unsafe model behavior before downstream action.
  • X. ETHICAL CONSIDERATIONS: Harmful prompts and outputs should be handled only as needed for evaluation, auditing, and reproducibility, while minimizing human exposure.This ethical constraint applies to the study’s engagement with harmful prompts, unsafe generations, and dual-use failure analysis.
  • X. ETHICAL CONSIDERATIONS: Analyzing existing defense failures may clarify attack surfaces for adversaries, creating a dual-use risk that motivates careful threat-model and defense reporting.The paper explicitly notes that clearer reasoning about attack surfaces can result from failure analysis.
  • X. ETHICAL CONSIDERATIONS: Reference resolution is treated as a security boundary because visually grounded inputs can produce subtle, difficult-to-detect, and consequential failures.The motivating inputs include screenshots, scanned documents, diagrams, and other localized visual content.
  • X. ETHICAL CONSIDERATIONS: Evaluating this defense requires addressing both harmful exposure and overblocking rather than treating safety as harmful-content suppression alone.The paper frames overblocking as potentially harmful when harmless or unusual inputs are suppressed or burdened.
Loading 2608.17234v1…