PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models

Tianshi Wang, Jingsong Wang, Fengling Li, Jingjing Li, Xin Li, Lei Zhu

arXiv:2610.20679v1cs.AIcs.CRcs.MM

TL;DR

Representation-based VLM safety defenses can separate harmful from benign inputs without establishing what a risk score means or when its evidence applies. CNRF constructs context-matched counterfactual evidence, estimates neighborhood support separately from risk, and selectively reviews responses. Across three VLMs, it reports five-attack macro DSR values from 86.06% to 89.30% while retaining benign-task utility.

  • Problem

    Representation-based defenses separate harmful and benign inputs, but separation alone does not establish what a risk score represents or when its evidence should be trusted.

    Source: §1 · p. 1

  • Method

    CNRF constructs context-matched benign-harmful counterfactual pairs as capability-related risk directions, estimates neighborhood support separately, and routes selected requests for same-victim response review.

    Source: §1 · p. 2

  • Results

    Across Qwen, Gemma, and InternVL, five-attack macro DSR is 88.57%, 86.06%, and 89.30%, respectively.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    CNRF improves harmful-request resistance while preserving benign-task responsiveness across the evaluated VLMs and jailbreak families.

    Source: §4.2 · p. 7, §5 · p. 10

Abstract

from arXiv · show

Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk. This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show that query-matched references and preserved context-direction correspondence materially improve risk ranking in the low-false-positive regime. CNRF thus reframes representation-based VLM safety from learning harmfulness boundaries to constructing applicable evidence for selective risk decisions.

1 Introduction

VLM safety must distinguish harmful capability requests from benign requests that share similar topics or visual contexts, but representation separation alone does not establish what risk scores mean or when they apply. CNRF constructs context-matched evidence and separates capability-related risk from evidence applicability to support selective safety decisions.

1 Introduction

  • VLM safety is difficult because benign and harmful requests can share topics, entities, and visual contexts, while multimodal jailbreaks can alter how intent is represented.Source: §1 · p. 1
  • Representation-based defenses can separate harmful and benign inputs, but separation alone does not explain what a risk score represents or when its evidence should be trusted.Source: §1 · p. 1
  • Boundary-based decisions can permit harmful assistance or reject benign requests with risk-related context, motivating applicable evidence for selective risk decisions.Source: §1 · p. 2, Figure 1 · p. 2
  • CNRF builds capability-related risk directions from context-matched benign-harmful pairs, estimates neighborhood support separately, and uses a victim-adaptive interface with same-victim response review.Review either releases acceptable candidates or generates constrained alternatives with reduced harmful capability.Source: §1 · p. 2
  • CNRF shifts representation-based safety from harmfulness-boundary learning toward evidence construction that separates capability-related risk from evidence applicability.Source: §1 · p. 2
  • CNRF is evaluated across three victim VLMs and five multimodal jailbreak families, with experiments validating query-matched references, preserved context-direction correspondence, and same-victim response review.Source: §1 · p. 3

2 Related Work

Prior work estimates safety from model representations, modifies inputs or responses, or intervenes on internal representations and generation. CNRF distinguishes itself by constructing capability-preserving counterfactual evidence and separating its applicability from risk.

Representation-based safety evidence

  • Representation-based defenses estimate safety from internal states; CNRF instead uses context-matched counterfactual capability transitions and estimates neighborhood support separately from risk.RCS uses class-relative distances, with KCD exploiting local neighbors and MCD modeling Gaussian class distributions.Source: §2 · p. 3

Input filtering and response-level safeguards

  • Input-modification and response-inspection approaches include defensive prompting, perturbation analysis, answer checking, and alignment; CNRF screens with evidence and reviews same-victim candidates.CNRF can release acceptable responses while constraining only responses that fail verification.Source: §2 · p. 3

Risk-aware internal intervention

  • Internal-intervention methods edit activations, attention, or safety subspaces, whereas CNRF enables selective routing and response review without directly modifying the victim model.Source: §2 · p. 3

3 Counterfactual Neighborhood Risk Fields

CNRF constructs context-matched evidence of capability transitions, estimates risk separately from reference support, and selectively reviews routed requests. Its screening interface adapts to the safety observations available from each VLM.

3 Counterfactual Neighborhood Risk Fields

  • CNRF frames representation-based VLM safety as constructing applicable evidence for selective risk decisions rather than deriving risk solely from harmful-benign separation.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Counterfactual Evidence Construction

  • Context-matched benign-harmful pairs preserve surrounding content while changing requested capability, making their representation differences evidence of capability transitions.Multiple renderings of a parent task provide within-label carrier variation.Source: §3.1 · p. 4

3.2 Capability Direction Modeling

  • Each pair’s midpoint anchors the comparison to its observed context, while its direction encodes the capability-related representation shift; endpoints are normalized to −1 and +1.Source: §3.2 · pp. 4–5

3.3 Neighborhood Risk Field

  • The neighborhood risk field combines directions from the nearest k distinct-parent midpoints, while a separate support estimate measures whether references cover the query.The intercept and fitted-benign statistics are used to standardize the risk estimate.Source: §3.3 · p. 5

3.4 Victim-Adaptive Screening

  • Victim-adaptive screening supports different safety observations across VLMs; neighborhood support reflects reference applicability, not harmfulness.For internal safety observations, the interface can use a victim-specific safety observer.Source: §3.4 · pp. 5–6

3.5 Same-Victim Response Review

  • Routed requests receive same-victim response review: acceptable candidates are released unchanged, while candidates failing checks receive constrained alternatives with reduced harmful capability.Separating inspection from refusal allows routed benign responses to pass verification rather than face blanket rejection.Source: §3.5 · p. 6

4 Experiments

Across three victim VLMs, CNRF combines strong defense outcomes with retained benign-task performance. Screening and controlled evidence-construction analyses examine risk ranking, routing, reference selection, and response review.

4.1 Experimental Setup

  • The evaluation covers three VLMs, five multimodal jailbreak families, and benign-task benchmarks, using a fixed CNRF configuration per victim across cohorts.Comparisons use representative representation-based detectors, end-to-end defenses, and risk-aware interventions under a unified input protocol and victim decoding configuration.Source: §4.1 · pp. 6–7

4.2 Safety and Utility Results

  • CNRF reaches five-attack macro DSRs of 88.57%, 86.06%, and 89.30% on Qwen, Gemma, and InternVL, respectively, leading on the first two.On InternVL, its macro DSR is approximately 0.12 points below OH-Defense while maintaining substantially better benign-task utility.Source: §4.2 · p. 7, Table 1 · p. 8
  • SUS is 69.82, 73.27, and 77.21 across the three victims; CNRF also retains MM-Vet scores of 63.25, 67.88, and 75.63 and MMMU accuracy within 2.67 points of native.MOSS non-refusal is 58.67%, 77.00%, and 70.33%; the lowest attack-family DSRs are 76.10% on Qwen MMJailBench and 77.65% on Gemma JOOD.Source: §4.2 · p. 7, Table 1 · p. 8

4.3 Representation Screening Analysis

  • CNRF achieves macro AUROC of 87.87%, 94.50%, and 92.74% and macro TPR@5% FPR of 60.76%, 80.75%, and 74.93% on Qwen, Gemma, and InternVL, exceeding evaluated baselines.It routes 74.55%, 72.44%, and 81.41% of attack units while benign direct passage rates are 92.50–100% on MM-Vet and 94.00–100% on MMMU.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 Evidence Quality Ablation

  • On 770 Qwen validation inputs, matched references yield pAUROC 89.24, versus 85.92 for global aggregation, 75.24 for random neighborhoods, and 48.59 for far neighborhoods.The fitted risk space and reliability weights are held fixed while reference selection changes.Source: §4.4 · p. 9, Figure 4 · p. 9
  • Replacing a matched capability direction with another parent’s direction reduces pAUROC from 89.24 to 75.32 and full-range AUROC from 94.14% to 77.25%.The controls preserve retrieval, midpoint, direction length, and reliability; matched AUROC on paired endpoints is 94.58% versus 91.12%.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 Same-Victim Response Review Analysis

  • In a fixed Qwen pilot, review releases 12 additional original responses across 48 benign inputs while macro DSR remains 75.42%.MOSS non-refusal rises from 56.25% to 62.50%, MM-Vet partial correctness from 35.00 to 58.13, and MMMU accuracy from 37.50% to 50.00%, without changing screening routes.Source: §4.5 · p. 10, Figure 5 · p. 10
  • On benign MM-Vet requests, CNRF adds 0.279–0.551 seconds mean latency and approximately 0.02–0.64 GiB maximum peak memory versus native generation.Against ECSO, mean latency is 0.059 and 0.073 seconds lower on Qwen and Gemma, respectively, and 0.090 seconds higher on InternVL.Source: §4.5 · p. 10, Figure 5 · p. 10

5 Conclusion and Limitations

The conclusion frames CNRF as evidence construction for selective VLM safety, with reference applicability and context-direction correspondence central to reliable risk estimates. The appendix states that evaluation settings are fixed before final evaluation.

5 Conclusion and Limitations

  • CNRF reframes representation-based VLM safety from harmfulness-boundary learning to constructing applicable evidence, combining capability comparisons, support estimation, and same-victim response verification.The reported analyses identify query-matched references and preserved context-direction correspondence as important to reliable evidence.Source: §5 · p. 10

A Experimental Setup and Implementation Details

  • Unless otherwise specified, hyperparameters, calibration procedures, and routing thresholds are determined using training- or development-role data and remain fixed during final evaluation.The appendix describes the datasets, evaluation protocols, implementation details, and prompts used in the experiments.Source: A Experimental Setup and… · p. 15

A.1 Data Sources and Evaluation Protocol

The evaluation uses benchmark cohorts with related examples kept within the same data partition, then measures safety, benign utility, and representation-screening performance under specified protocols.

A.1 Data Sources and Evaluation Protocol

  • Counterfactual endpoints, related rewrites, and shared-image groups stay within one fitting, calibration, or validation partition to prevent information leakage.Evaluation results characterize the selected benchmark suite; an independently sampled prospective evaluation remains a separate requirement.Source: §A.1 · p. 15
  • The panel contains 3,159 inputs per victim, and attack averages weight the five jailbreak families equally.Table 2 summarizes the evaluation cohorts and aggregation units.Source: Table 2 · p. 15
  • Counterfactual rewrites preserve contextual factors while changing requested behavior along six axes, and candidates are filtered for overlap, residual harmful operations, image-dependence changes, axis inconsistency, and duplication.The attack cohorts comprise five multimodal jailbreak families, while benign utility cohorts use MOSSBench, MM-Vet, and MMMU.Source: §A.1 · pp. 15–16

A.2 Metrics and Evaluation Settings

  • Benign evaluation distinguishes refusal behavior, partial-credit multimodal capability, and exact-choice accuracy, while attack metrics include ASR3 as a sensitivity criterion using s ≥3.MOSSBench counts full or partial refusal as failure; MM-Vet and MMMU score partial correctness and exact-choice correctness, respectively.Source: §A.1 · p. 16, §A.2 · p. 16
  • Automated judgments may inherit biases from the shared model family and should be interpreted separately from human assessments.Unknown outcomes retain their denominators, with conservative bounds for attack/failure metrics and correctness metrics.Source: §A.2 · p. 16
  • Representation screening reports AUROC, TPR at 5% FPR, and standardized partial AUROC over FPR 0–10%, with equal-weight benign distributions and attack-family averaging.For JOOD, the minimum score across three attempts is used, and a goal is routed only when all three attempts meet the routing criterion.Source: §A.2 · p. 16

A.3 CNRF Implementation Details

CNRF adapts observation and routing to each victim, builds risk and support from local representation geometry, and uses response review to release or constrain candidates.

A.3 CNRF Implementation Details

  • Qwen and InternVL use local readers, while Gemma uses an internal safety observer; Table 3 summarizes local-reader settings.Each observation comprises four relative depths and two token readouts, and α determines a benign calibration quantile.Source: §A.3 · p. 17, Table 3 · p. 17
  • Qwen selects projection directions using risk-to-nuisance criteria, whereas InternVL uses signal-only geometry; both compute projected midpoints, directions, and benign support.Qwen uses a paired-midpoint intercept, while InternVL centers using local benign references; parent-weighted logistic fitting excludes references from the query’s own parent.Source: §A.3 · p. 17
  • Support measures standardized distance to the kth distinct-parent benign neighbor, while the priority operator adapts to victim observations and InternVL limits support uplift.Qwen’s support state is diagnostic and does not independently trigger review; a near-zero aggregate direction authorizes no representation edit.Source: §A.3 · p. 17
  • Response verification uses a separate evidence bank and conjunctive risk and support conditions, then a retrieved behavioral axis constrains alternatives when candidates fail review.InternVL’s coverage-conditioned release requires a further same-victim capability check below 3; visual fallback generation omits the original image and source instruction.Source: §A.3 · p. 18
  • Across 9,477 evaluation inputs, native candidates, answers, and routes remain unchanged, with detector scores agreeing within 10^-7.Execution reuse limits image-feature caching to two entries per request and transfers only required hidden-state rows to the CPU.Source: §A.3 · p. 18

A.4 Evaluation Prompts

  • The evaluation prompts classify benign responses as full compliance, full refusal, or partial refusal, and judge attacks by whether responses materially help the harmful objective.Attack scores distinguish no usable assistance, partial assistance, actionable assistance, and exhaustive or optimized assistance.Source: §A.4 · pp. 18–19

B Baselines and Experimental Protocols

  • Baseline comparisons retain original method names and released implementations or published specifications, with architecture-dependent adaptations documented for reproducibility.Comparisons use the same victim inputs, outer decoding settings, and cohort-specific judges unless otherwise specified.Source: B Baselines and Experimental Protocols · p. 19

B.1 Baseline Implementations

The baseline implementations preserve released procedures where possible, while adapting fitting data, validation, or architecture-specific settings to support controlled comparisons.

B.1 Baseline Implementations

  • HiddenDetect retains its refusal tokens, final-token states, normalization, head, and layer-selection criterion, but Qwen uses a raw score because only one valid layer is selected.Binary thresholds use calibration examples.Source: §B.1 · p. 19
  • RCS-KCD and RCS-MCD retain the released scoring procedures but are fitted on CNRF’s counterfactual archive, so results compare scoring under shared supervision rather than reproduce original benchmark numbers.KCD uses k = 40 following evaluated code, whereas the published paper specifies k = 50.Source: §B.1 · pp. 19–20
  • VLMGuard follows its manuscript specification, while a separate 50-harmful/50-benign validation set selects its threshold by balanced accuracy with tie-breaking rules.Its implementation uses centered SVD, variance-weighted top-k pseudo-labeling, and a two-hidden-layer classifier.Source: §B.1 · p. 20
  • ECSO and CIDER retain their specified response-checking and perturbation procedures, with victim-specific wrappers for ECSO and a fixed auxiliary encoder for CIDER.CIDER inputs without images bypass its detector.Source: §B.1 · p. 20
  • ASTRA follows its released-code configuration of 64 ablations and every-forward intervention rather than the manuscript’s 96-ablation, generated-token-only setting.Its layers and nonzero strengths are selected on training data under a five-percentage-point benign degradation constraint.Source: §B.1 · p. 20
  • OH-Defense uses reported Qwen attention heads and scans all Query heads for Gemma and InternVL, while CARE retains its projection, attribution, and intervention procedures.CARE uses a thin singular-value/Woodbury formulation for projection.Source: §B.1 · p. 20
  • HiddenDetect fitting avoids JailbreakV evaluation-source overlap by using disjoint benign and harmful training sources.AdaShield-A retains its released split, refinement rounds, defender, retrieval settings, and victim-specific prompt pools; pool/test image overlap and cross-victim reuse are excluded.Source: §B.1 · p. 20

B.2 Controlled Experimental Protocols

The controlled protocols vary reference selection, direction correspondence, geometry, response actions, and efficiency conditions while fixing specified model and routing components.

B.2 Controlled Experimental Protocols

  • Matched retrieval selects 16 nearest distinct parents, while random and far controls select the same number independently of the query or by maximum distance.Global aggregation uses the complete reference bank with equal total weight per parent.Source: §B.2 · p. 21
  • Direction-only mismatch swaps a parent’s unit direction within the same carrier while preserving midpoint, chord length, reliability, and retrieval position.Random and mismatch results average ten seeds, and the paired-endpoint subset contains 430 inputs from 55 parents.Source: §B.2 · p. 21
  • Geometry controls replace local aggregation with a global affine reader, set reliability weights equally, or reassign directions with weights and norms within carriers.These interventions keep model parameters and deployment thresholds fixed and assess fitted components without retraining alternative models.Source: §B.2 · p. 21
  • Response-action controls compare native generation, fixed refusal, generic constrained review, and full review on 3,159 inputs per victim with routing decisions held identical across non-native arms.Generic review removes response-bank verification and counterfactual conditioning, isolating their joint contribution; unrouted answers and fixed hard alarms remain unchanged.Source: §B.2 · p. 21
  • A separate Qwen review pilot uses 130 route-stratified inputs and varies reviewer reference information while keeping routes, candidates, and fallback responses fixed.Approved candidates are released unchanged; otherwise the existing CNRF response is retained.Source: §B.2 · p. 21
  • Benign efficiency measurements compare native generation, ECSO, and full CNRF on the same 32 MM-Vet requests per victim, reporting per-request latency and peak allocated memory.Timing includes the complete online call but excludes model and asset loading and external judging; the measured subset has no ECSO conditional-path activations.Source: §B.2 · p. 22

C Additional Experimental Results

  • The additional-results section covers extended end-to-end benchmark comparisons and representation-level screening through risk ranking and deployment-oriented routing.Bootstrap intervals condition on selected cohorts and fixed judges, and shared source tasks or images may introduce additional dependence.Source: §B.2 · p. 21, C Additional Experimental Results · p. 22

C.1 External Main Results

The appendix extends evaluation to external safety, robustness, and utility benchmarks and examines representation screening and its mechanisms. Controls support correspondence-aware reference selection, while the local–global comparison does not establish a clear winner.

C.1 External Main Results

  • Table 4 adds end-to-end external-benchmark results covering safety and benign utility under the same victim models, cohorts, and protocols.The table reports attack defense success rate and benign measures including MOSS non-refusal, MM-Vet partial correctness, and MMMU exact-choice accuracy.Source: §C.1 · p. 22, Table 4 · p. 23

C.2 Representation Detection and Routing

  • Table 5 reports continuous-score ranking and deployed routing statistics, distinguishing low-FPR detector quality from benign direct-passage trade-offs.Its AUROC, TPR@5% FPR, and standardized pAUC0:10 average five attacks against equally weighted MOSS, MM-Vet, and MMMU benign distributions.Source: §C.2 · p. 22, Table 5 · p. 24

D Mechanistic Analysis and Response Review

  • The appendix analyzes how counterfactual correspondence and local geometry affect representation screening, then evaluates response-review mechanisms for selective answer recovery.Table 6 contains fixed-reader controls, and Table 7 gives paired AUROC contrasts on the complete validation pool and paired-endpoint subset.Source: D Mechanistic Analysis and Response… · p. 22, §D.1 · p. 23

D.1 Counterfactual Correspondence and Reference Selection

  • Matched references improve ranking over random retrieval, far retrieval, and direction-mismatched evidence; the local–global AUROC difference is smaller and its confidence intervals span zero.The results support correspondence-aware evidence selection under the fitted reader, not a uniquely optimal aggregation strategy.Source: §D.1 · p. 23, Table 6 · p. 24, Table 7 · p. 24

D.2 Geometry, Locality, and Carrier Analysis

Geometry and carrier controls show that evidence behavior varies with model and carrier composition. Response review can recover selected benign answers, with measurable, model-dependent safety–acceptance trade-offs.

D.2 Geometry, Locality, and Carrier Analysis

  • Qwen controls depend more strongly on original reference alignment, while InternVL reassignment overlaps with and can exceed matched-reference scores.Table 8 reports post-fit AUROC interventions; Gemma uses an observer-only routing operator and is tested separately.Source: §D.2 · p. 25, Table 8 · p. 25
  • Novelty predicts risk strongly with benign-only carriers but becomes less informative on the paired subset, separating evidence applicability from risk scoring.Local–global ordering varies by carrier composition: the local reader performs better on the complete pool, while the global reader performs better on the two-label-carrier subset.Source: §D.2 · p. 25, Table 9 · p. 25
  • Gemma’s detector is observer-based: removing the local field leaves all 3,159 routes unchanged, whereas restoring it without the observer changes 1,254 routes.Its results therefore characterize observer-based screening, not the local-coordinate geometry used by Qwen and InternVL.Source: §D.2 · p. 25
  • The local-coordinate error bound decomposes into direction mismatch, contextual-intercept mismatch, and residual representation error.Locality can reduce intercept mismatch, but directional correspondence must be evaluated independently; aggregate behavior and carrier dependence are evaluated empirically.Source: §D.2 · p. 25

D.3 Response Review and Candidate Verification

  • Full review releases 13 additional benign MOSS responses on InternVL, reducing refusal failure by 4.33 points versus generic review while raising macro ASR by 0.77 points.MM-Vet and MMMU scores remain the same; Qwen and Gemma show no differences between full and generic review in macro ASR or reported benign metrics.Source: §D.3 · p. 26, Table 10 · p. 26, Table 11 · p. 26
  • In the fixed 130-input Qwen pilot, candidate review selected 12 newly released benign buffers (2 MOSS, 4 MM-Vet, and 6 MMMU), while macro ASR remained 24.58%.MOSS non-refusal rose from 56.25% to 62.50%, MM-Vet from 35.00 to 58.13, MMMU accuracy from 37.50% to 50.00%, and SUS from 55.65 to 65.56; this selected-cohort experiment does not establish full-panel performance or a card-specific advantage.Source: §D.3 · p. 26

E Qualitative Case Studies

The selected case studies illustrate candidate release when answers address benign or preventive interpretations despite risk-associated context or ambiguous wording. The examples illustrate decisions, not their frequency or correctness across the evaluation set.

E Qualitative Case Studies

  • The eight InternVL examples are outcome-selected recovered non-refusals, so they illustrate release decisions rather than their frequency, correctness, or an isolated counterfactual-card effect.Available records support text comparisons, not reconstructed images or inferred risk scores.Source: E Qualitative Case Studies · p. 27
  • The released birthday-party answer addresses party organization despite a toy gun in the image.The case demonstrates a benign request handled in risk-associated visual context.Source: E Qualitative Case Studies · p. 27
  • A living-room request receives advice about using the seating area rather than operating depicted firearms.The recorded response describes firearm-themed wall art in the room.Source: E Qualitative Case Studies · p. 27
  • A request to write a policy brief with child-labor imagery receives a response advocating measures to protect children from exploitation.The case is labeled protective policy writing.Source: E Qualitative Case Studies · p. 27
  • For the ambiguous request about encouraging something in a drunk-driving scene, the released answer advocates responsible behavior and discourages DUI.The interpretation is a property of the candidate, not evidence that the input wording is unambiguous.Source: E Qualitative Case Studies · p. 27
  • Other released candidates address anti-trafficking education, chess strategy, traffic guidance, and the relational meaning of figurative wording.These examples interpret the request as prevention-oriented, game-related, ordinary guidance, or relational rather than capability-enhancing or literal harm.Source: E Qualitative Case Studies · p. 28

Related papers

Loading 2610.20679v1…