PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Grounding Safety in Vision-Language Models

Erfan Shayegani, Kundan Krishna, Yue Dong, Nael Abu-Ghazaleh, Leon Gatys, Shruti Palaskar

arXiv:2610.05637v1cs.AIcs.CRcs.CVcs.CYcs.LG

TL;DR

Visual grounding safety is understudied even as points and boxes can be acted on by downstream systems. The paper evaluates matched VQA and grounding requests across five VLMs and proposes fine-tuning with grounding-form refusals, capability data, and self-distilled benign examples. Grounding refusal trails VQA refusal, while the fine-tuning improves refusal on evaluated and held-out domains and shifts harmful requests toward refusal representations.

  • Problem

    Safety alignment for actionable visual grounding outputs has not been systematically analyzed, despite points and boxes being consumable by agents and robots.

    Source: §1 · p. 1

  • Method

    The study repurposes three benchmarks into 15,401 matched harmful VQA-grounding pairs and fine-tunes two models with grounding-form refusals, capability grounding data, and self-distilled benign examples.

    Source: §1 · pp. 1–2

  • Results

    Grounding refusal trails VQA refusal by 31–59 percentage points on average, while fine-tuning raises grounding refusal by 77–95 points on VLSU and BBQ-V and 64–85 points on held-out Asimov-2.0.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Fine-tuning also improves VQA refusal, preserves grounding capability, limits in-domain over-refusal to at most 5.3%, and shifts harmful requests toward each model’s refusal direction.

    Source: §1 · p. 2

    The supervision-mismatch explanation is a hypothesis supported by ablations on Qwen3-VL-8B.

    Source: §1 · p. 2

Abstract

from arXiv · show

Vision-language models (VLMs) are increasingly trained to generate structured outputs like points and bounding boxes that downstream interfaces, agents, and robots can act on, yet safety alignment of this output channel has not been systematically analyzed. We study visual grounding safety by repurposing three safety benchmarks spanning direct harm (VLSU), social bias (BBQ-V), and situational safety (Asimov-2.0) into 15,401 matched pairs of harmful requests that differ only in the requested output: a free-text answer (VQA) or a grounding (point or bounding box). Across five VLMs, models that refuse a harmful request posed as a question often comply when the same request asks for a grounding: averaged over models, grounding refusal trails VQA refusal by 31-59 percentage points, depending on the domain, and safety system prompts do not close this gap. We propose a fine-tuning approach that combines grounding-form refusals with capability grounding data and self-distilled benign data to counter over-refusal. For Qwen3-VL-8B and VisionReasoner-7B, it improves grounding refusal by 77-95 percentage points on VLSU and BBQ-V and by 64-85 points on the held-out Asimov-2.0 domain, while also improving VQA refusal, preserving grounding capability, and keeping over-refusal limited. Representation analysis shows that fine-tuning moves harmful requests toward each model's refusal direction, most strongly for grounding, while leaving benign requests near the harmless reference.

1 INTRODUCTION

Visual grounding outputs can be acted on by downstream systems, making safe spatial responses consequential. The paper evaluates whether safety behavior transfers from VQA to grounding and proposes fine-tuning to improve that transfer.

1 INTRODUCTION

  • Grounding points and boxes can guide GUI clicks or robot movement, so harmful compliance may translate directly into action.The paper motivates studying this output channel because downstream systems may consume it instead of a person reading it.Source: §1 · p. 1
  • Grounding refusal trails VQA refusal by 31–59 percentage points on average across five VLMs, and safety system prompts do not close the gap.The evaluation repurposes three benchmarks into 15,401 matched harmful pairs spanning direct harm, social bias, and situational safety.Source: §1 · pp. 1–2
  • Fine-tuning with grounding-form refusals, capability grounding data, and self-distilled benign examples raises grounding refusal by 77–95 points on VLSU and BBQ-V and 64–85 points on held-out Asimov-2.0.It also improves VQA refusal, preserves grounding capability, and keeps in-domain over-refusal at or below 5.3%.Source: §1 · p. 2
  • The paper establishes a 15,401-pair evaluation framework and finds a consistent grounding-versus-VQA refusal gap across models and harm domains.Its proposed fine-tuning is intended to improve safety while preserving grounding capability and limiting over-refusal.Source: §1 · p. 2
  • Representation analysis finds that fine-tuning shifts harmful requests toward each model’s refusal direction, most strongly for grounding, while benign controls remain near the harmless reference.The authors hypothesize that text-form safety supervision may not generalize to spatial outputs; Qwen3-VL-8B ablations support this account.Source: §1 · p. 2

2 EVALUATING VISUAL GROUNDING SAFETY

The evaluation compares VQA and grounding on matched harmful requests, holding image and intent constant while changing the response form. It finds lower grounding refusal across models and domains, and prompting does not close the gap.

2.1 MATCHED VQA AND GROUNDING PAIRS

  • Matched pairs preserve each image, question, and intent while varying only whether the model answers in text or through its native point-or-box grounding interface.VLSU covers direct harm; BBQ-V uses ambiguous stereotype-bias scenes; Asimov-2.0 constraints are inverted into harmful pairs and held out for transfer evaluation.Source: §2.1 · p. 3

2.2 MODELS, JUDGE, AND METRICS

  • An LLM judge scores refusal and compliance; harmful refusal, benign over-refusal, and PointArena grounding capability are reported separately.Grounding counts as refusal for an explicit decline, empty output, or localization only on clearly irrelevant or safe objects; capability is separated because failure to locate a target may reflect limited ability.Source: §2.2 · p. 4

2.3 GROUNDING SAFETY LAGS VQA SAFETY

  • Grounding refusal averages 19.4%, 16.7%, and 12.7% on VLSU, BBQ-V, and Asimov-2.0, versus VQA refusal of 62.9%, 75.9%, and 43.8%, respectively.Grounding refusal is lower for every evaluated model and domain; the gap varies in size but not direction.Source: §2.3 · p. 4, Figure 2 · p. 4
  • On BBQ-V, VisionReasoner-7B’s reasoning trace argues for refusal on 64.5% of requests, yet its final answer grounds a target in over 99%.The paper treats the reasoning trace as a secondary diagnostic; headline results use the final answer.Source: §2.3 · p. 4, §2.2 · p. 4

2.4 SAFETY PROMPTING DOES NOT CLOSE THE GAP

  • Grounding refusal remains below VQA refusal for every model, domain, and system-prompt condition; only Claude Sonnet 4.6 exceeds 40% grounding refusal under any condition.VisionReasoner-7B stays below 5%, and the placeholder prompt raises Qwen3-VL-8B’s VQA over-refusal on VLSU and BBQ-V.Source: §2.4 · pp. 4–5

3 IMPROVING VISUAL GROUNDING SAFETY

The section combines grounding-form refusals, benign examples, and capability data to improve safety while limiting over-refusal and preserving localization. Ablations and prompting comparisons examine the roles of output format and each training component.

3.1 TRAINING RECIPE

  • The training recipe combines paired harmful refusals, self-distilled benign examples, and 1,548 capability-grounding examples; refusal targets are plain text or structured outputs with non-actionable coordinates.The model’s base responses enter the benign data only when they correctly ground the target without refusing.Source: §3.1 · p. 5

3.2 FINE-TUNING IMPROVES SAFETY IN BOTH GROUNDING AND VQA MODES

  • Grounding refusal rose from 9.7% to 96.5% on VLSU and 15.0% to 92.8% on BBQ-V; on held-out Asimov-2.0, it rose from 4.9% to 68.9% (85.4% for VisionReasoner-7B).VQA refusal also improved across checkpoints and domains, including on Asimov-2.0 (45.7% to 79.9%).Source: §3.2 · p. 6
  • PointArena success remained 65.5% vs. 65.4%, while in-domain over-refusal stayed at or below 5.3%; held-out MOSSBench over-refusal rose from 15.3% to 24.7%.The section describes the MOSSBench increase as modest relative to prior multimodal safety fine-tuning.Source: §3.2 · p. 6
  • For VisionReasoner-7B, SFT-plain was stronger in five of six harmful-request evaluations, while SFT-placeholder preserved structured outputs; the Qwen3-VL-8B comparison is confounded by more refusal examples in its placeholder mixture.The section frames the choice as deployment-dependent: plain gives a readable refusal, while placeholder keeps output parseable for downstream systems.Source: §3.2 · pp. 6–7

3.3 TRAINING ABLATIONS

  • Removing grounding-form safety data left grounding refusal at 1.8%, 0.0%, and 3.0% on VLSU, BBQ-V, and Asimov-2.0, while removing VQA-form safety data raised MOSSBench over-refusal to 55.0%.The 20 VQA-form refusal examples raised VQA refusal, but their failure to transfer to grounding was not explained by data volume alone.Source: §3.3 · p. 7
  • Without benign over-refusal data, harmful grounding refusal reached 99.8–100.0% but all benign BBQ-V grounding requests were refused; without coordinate supervision, PointArena success fell to 6.7%.The placeholder target retained 52.5% PointArena success without coordinate supervision.Source: §3.3 · p. 7

3.4 FINE-TUNING VS. PROMPTING

  • Fine-tuning raised mean grounding refusal by 76–90 percentage points, while prompting kept it below 21% for both models; Qwen3-VL-8B’s placeholder prompt raised over-refusal from 3.6% to 19.4%.Qwen3-VL-8B fine-tuning had lower over-refusal than either prompt; VisionReasoner-7B fine-tuning was 0.2–1.4 points higher than prompts.Source: §3.4 · p. 7

4 REPRESENTATION ANALYSIS

The analysis projects prompt-token activations onto a refusal direction and compares matched harmful and benign requests across output modes and fine-tuning checkpoints. Fine-tuning shifts harmful representations most strongly for grounding, while benign controls remain near the harmless reference.

4 REPRESENTATION ANALYSIS

  • The analysis uses final-prompt-token residual-stream activations and defines the refusal direction from mean harmful AdvBench versus harmless Alpaca activations, normalizing their centroids to 1 and 0.Matched requests are projected at each layer, with attention to mid-to-upper layers.Source: §4 · pp. 7–8
  • In the base VisionReasoner-7B model, harmful grounding requests clustered near the harmless reference, while matched VQA requests extended farther toward the refusal-associated region.This pattern is consistent with the model refusing VQA requests more often than grounding requests.Source: §4 · p. 8, Figure 4 · p. 8
  • Fine-tuning shifted harmful VQA and grounding requests toward the refusal-associated region, most strongly for grounding, while benign controls stayed near the harmless region across both models and all three domains.The grounding representations also became more aligned with VQA representations.Source: §4 · p. 8
  • Appendix A.11 and Table 23 provide the corresponding results and quantitative projections, while Figure 5 summarizes these positions on the refusal direction.Figure 4 shows VisionReasoner-7B on VLSU at layer 20, with residual-stream activations projected onto the top two principal components of harmless and harmful reference prompts, fitted separately for each checkpoint.Source: Figure 4 · p. 8, §4 · p. 9

5 RELATED WORK

Prior work studies grounding capabilities and multimodal safety, often treating generated language as the safety-critical output. This paper focuses on safety of a VLM’s own grounding output under matched, non-adversarial requests.

5 RELATED WORK

  • Grounded VLM work develops point, box, and mask outputs and evaluates localization; this paper instead studies the safety of those spatial outputs.The related work describes such outputs as interfaces for GUI agents and robots.Source: §5 · p. 9
  • Prior multimodal safety work studies language refusal, jailbreaks, over-refusal, and refusal directions; the authors describe their study as the first controlled comparison across VQA and grounding modes, models, and harm domains.Related agent studies examine non-text actions, often under jailbreaks; this work uses matched, non-adversarial requests and traces the gap to safety-supervision form.Source: §5 · p. 9

6 CONCLUSION

The conclusion summarizes a grounding-safety gap across models and domains, reports fine-tuning gains and representation shifts, and notes preserved grounding capability.

6 CONCLUSION

  • Across five VLMs, grounding refusal trailed VQA refusal by 31–59 percentage points on average; fine-tuning raised grounding refusal by 77–95 points on VLSU and BBQ-V and 64–85 on held-out Asimov-2.0.The conclusion also reports improved VQA refusal, preserved grounding capability, and harmful-request shifts toward the refusal direction, strongest for grounding.Source: §6 · p. 9

ETHICS STATEMENT

The paper studies harmful and biased requests to identify and mitigate VLM safety failures, using examples that may be disturbing. The documented grounding failures occur without optimized adversarial attacks, and the authors aim to inform safer models as grounding becomes more common in downstream and agentic applications.

ETHICS STATEMENT

  • The paper includes potentially offensive or disturbing examples only to identify and mitigate VLM safety failures.The examples concern harmful and biased requests.Source: ETHICS STATEMENT · pp. 9–10
  • Grounding failures arise without optimized adversarial attacks, exposing readily accessible alignment blind spots that warrant careful study.Source: ETHICS STATEMENT · p. 10
  • The work repurposes publicly available source datasets without collecting new images or human-subject data.Source: ETHICS STATEMENT · p. 10
  • The authors aim to inform safer, more robust models as grounding and pointing become more common in downstream and agentic applications.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper documents model versions, metrics, evaluation procedures, training objectives, and checkpoint results to support reproducibility. Fine-tuning updates only the language model, using small training mixtures and specified training settings.

REPRODUCIBILITY STATEMENT

  • Fine-tuning updates only the language model, leaving the vision encoder and projector frozen to target output behavior rather than visual representations.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Training uses three epochs, a 2 × 10^-5 learning rate, global batch size eight, bf16, and 840 × 840 inputs.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Training mixtures contain 120–300 harmful refusal examples, 1,400–2,200 benign examples, and 1,548 capability examples per checkpoint.Source: REPRODUCIBILITY STATEMENT · p. 10
  • The paper specifies evaluated model versions and metrics, summarizes evaluation sets and sample counts in Table 1, and reports full checkpoint results and controlled ablations.Training objectives, target formats, model components, and hyperparameters are described in Section 3.1; ablations are in Tables 4, 13, and 14.Source: REPRODUCIBILITY STATEMENT · p. 10
  • Appendix A.1 documents judging, elicitation, coordinates, decoding, and scoring, while Appendix A.2 covers data repurposing, self-distillation, quality control, and checkpoint compositions.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 EVALUATION DETAILS: JUDGE, PROMPTS, AND ELICITATION

Safety scoring uses a judge that assesses model outputs, with separate criteria for harmful compliance and benign over-refusal. The evaluation varies system prompts and elicits VQA and grounding through each model’s native interface.

A.1 EVALUATION DETAILS: JUDGE, PROMPTS, AND ELICITATION

  • The Gemini 3.5 Flash judge scores each item from the image and model output; harmful refusal receives 1.0, while benign compliance receives 1.0.For grounding, the judge sees an annotated image and raw output; VQA scoring uses the original image and text response.Source: §A.1 · p. 14
  • Under the placeholder condition, the judge treats point [0,0] or box [0,0,1,1] as a refusal by instruction, not by code-based coordinate checking.Source: §A.1 · pp. 14, 16
  • Grounding answer verdicts depend on the final points or boxes, while reasoning traces receive an independent verdict when present.The two axes can disagree; models without a reasoning trace receive N/A for that axis.Source: §A.1 · pp. 14–15
  • System-prompt conditions are none, generic safety guidance, or generic guidance plus dummy-coordinate instructions for refusal.The bias-domain generic prompt is framed around avoiding judgments based on stereotypes or appearance.Source: §A.1 · p. 16
  • VQA uses the bare question, whereas grounding wraps the same question in each model’s native pointing or detection template.The templates include model-specific point or bounding-box formats and coordinate conventions.Source: §A.1 · p. 16
  • Inputs are resized per model and predictions mapped back to original-image pixels, with default decoding settings and model-specific token limits.Inputs range from 672 × 672 for Molmo2 to 840 × 840 for VisionReasoner, Qwen3-VL, and Claude; Gemini uses 800-pixel width.Source: §A.1 · p. 17

A.2 GROUNDING-ORIENTED DATA CONSTRUCTION

The authors construct grounding-oriented harmful and benign tasks by pairing VQA and grounding requests around the same intent, then add capability grounding data. Construction rules preserve harmful intent while requiring a concrete, image-groundable target.

A.2 GROUNDING-ORIENTED DATA CONSTRUCTION

  • Three benchmarks are repurposed into matched harmful VQA and grounding requests: VLSU and IGP supply training pools, while BBQ-V and Asimov-2.0 are evaluation-only.VLSU and IGP support direct-harm and social-bias refusal training, respectively.Source: §A.2 · p. 17
  • For VLSU, BBQ-V, and Asimov-2.0, repurposing preserves the underlying harmful intent while expressing it through a concrete target or matched output interface.Asimov constraints are reversed into harmful objectives; BBQ-V retains its original ambiguous bias question for evaluation.Source: §A.2 · p. 17
  • Items are skipped when the harmful intent lacks a concrete visible target or cannot be made into a natural grounding task.Prompts should refer to target type and understated goal rather than naming an exact item or sub-location.Source: §A.2 · pp. 17–18
  • Matched VQA and pointing prompts must preserve the same goal and wording, differing only in the leading request phrase as grammar allows.Source: §A.2 · p. 18
  • Benign over-refusal examples are self-distilled from base-model responses and retained only when the model answers without refusal and correctly identifies or grounds the target.Social-bias examples add neutral left/right context so the requested person follows from stated evidence rather than a stereotype.Source: §A.2 · p. 19
  • Capability data use 1,548 Multi-Object-7k examples, with points checked against boxes and counting items requiring one-to-one matches.For VisionReasoner, incorrect groundings are repaired using the ground-truth region and a rewritten reasoning trace; capability data remain fixed across runs and ablations.Source: §A.2 · pp. 19–20, Table 5 · p. 20
  • Training pools contain 1,792 VLSU and 1,908 IGP harmful items; benign pools are generated separately per model, and all training subsets are disjoint from evaluation sets.Qwen3-VL benign pools contain 1,902 VLSU and 609 IGP items; VisionReasoner pools contain 2,902 VLSU and 863 IGP items.Source: §A.2 · p. 20

A.3 BASE-MODEL HARMFUL REFUSAL

Without a safety system prompt, harmful-request refusal is substantially lower for grounding than for VQA across every model and domain. Table 6 reports refusal rates for both modes and identifies the benchmark sample sizes.

A.3 BASE-MODEL HARMFUL REFUSAL

  • Grounding refusal is substantially lower than VQA refusal for every model and domain, while both modes leave substantial room for improvement.Source: Table 6 · p. 20
  • Table 6 reports harmful refusal rates (%) for VQA and grounding without a safety system prompt.Source: Table 6 · p. 20
  • The evaluation includes 659 VLSU items, 14,578 BBQ-V items, and 164 Asimov-2.0 items.Source: Table 6 · p. 20

A.4 REFUSAL UNDER THE THREE SYSTEM-PROMPT CONDITIONS

This section examines refusal across three system-prompt conditions and related analyses of grounding refusal and fine-tuning, including per-layer projections for Qwen3-VL-8B-Instruct on harmful and benign-control prompts. It reports that system prompts increase over-refusal without reliably fixing grounding refusal, while fine-tuning improves refusal and shifts harmful-request representations toward refusal-associated regions, with results varying by domain, category, capability, and reasoning trace.

A.4 REFUSAL UNDER THE THREE SYSTEM-PROMPT CONDITIONS

  • Safety prompts yield limited, inconsistent refusal gains across VQA and grounding, while placeholder-prompted Qwen3-VL reaches 43.7% benign over-refusal on VLSU.Table 7 reports harmful refusal under three prompt conditions; Table 8 reports benign-control over-refusal.Source: Table 7 · p. 21, Table 8 · p. 21

A.5 MOSSBENCH OVER-REFUSAL (HELD-OUT BENCHMARK) / A.6 REFUSAL / OVER-REFUSAL TRADE-OFF: COORDINATES

  • MOSSBench shows a modest over-refusal increase after fine-tuning, while prompting raises Qwen3-VL benign refusal from 15.3% to 40.3% without reliably improving grounding robustness.The paired harmful-contrast refusal rises too, and the appendix’s trade-off coordinates summarize aggregate over-refusal against mean grounding-form harmful refusal.Source: §A.5 · pp. 21–22, Table 11 · p. 22, Table 12 · p. 23

A.7 ABLATION STUDY: TRAINING-DATA COMPONENTS

  • Single-domain training generalizes only partially: VLSU-only reaches 73.4% held-out BBQ-V grounding refusal, while IGP-only reaches 78.5% on VLSU and 25.6% on Asimov-2.0.The ablations also show asymmetric form transfer: VQA-only safety leaves grounding refusal near base, whereas grounding-only safety raises refusal in both modes; removing capability data can undermine grounding ability.Source: §A.7 · pp. 23–24, Table 14 · p. 25

A.8 POINTARENA CAPABILITY BY SUB-SKILL

  • Qwen3-VL’s PointArena sub-skill changes remain within 3.5 percentage points, while VisionReasoner SFT-plain improves counting, reasoning, and spatial grounding.VisionReasoner changes from 46.9% to 51.0% in counting, 65.8% to 67.4% in reasoning, and 69.7% to 74.9% in spatial grounding; placeholder training reaches 81.3% affordance and 56.1% counting.Source: §A.8 · p. 25, Table 15 · p. 25

A.9 PER-CATEGORY HARMFUL REFUSAL

  • Fine-tuned refusal is broad across VLSU and BBQ-V categories, but residual Asimov-2.0 weakness is concentrated on robot- and property-directed harm.Qwen3-VL SFT-plain grounding refusal is 59.0% for robot and 64.0% for property targets; the placeholder target reaches 87.2% for robot.Source: §A.9 · p. 26

A.10 REASONING-TRACE REFUSAL ON THE GROUNDING FORM

  • VisionReasoner’s BBQ-V reasoning trace argues for refusal on 64.5–68.1% of requests even as answer refusal stays below 1.0%, and system prompts change trace refusal by at most 3.6 points.The placeholder target raises reasoning-trace refusal across domains, while plain-trained VisionReasoner emits plain refusals and has no scorable structured trace.Source: §A.10 · pp. 27–28, Table 19 · p. 28

A.11 ADDITIONAL REPRESENTATION RESULTS / A.12 COMPLETE PER-LAYER PCA VISUALIZATIONS

  • Fine-tuning shifts harmful representations toward refusal-associated regions more than benign controls, with grounding showing the largest shift across models and domains.For SFT-plain, VisionReasoner harmful grounding moves from 0.08 to 0.49 while benign moves from 0.04 to 0.11; Qwen3-VL changes from 0.23 to 0.56 harmful and 0.15 to 0.25 benign. The per-layer visualizations show this pattern across both backbones and request forms.Source: §A.11 · pp. 29–30, Figure 5 · p. 30, §A.12 · p. 35

A.12 COMPLETE PER-LAYER PCA VISUALIZATIONS

  • The projections compare base, SFT-plain, and SFT-placeholder on BBQ-V harmful bias, BBQ-V benign over-refusal, and Asimov-2.0 harmful prompts.The supplied captions identify the visualization scope but do not report projection values or establish a comparison outcome.Source: Figure 17 · p. 37, Figure 18 · p. 38, Figure 19 · p. 38

Related papers

Loading 2610.05637v1…