MLLMs Fail to Refuse when Using Tools Agentically

1 Introduction

• The paper identifies tool use as an overlooked safety blind spot: harmful requests that non-agentic MLLMs typically refuse are less likely to be refused with tools.The example pairs a harmful request with a relevant image; refusal safety has been studied extensively, but this tool-use vulnerability had largely gone unnoticed.Source: §1 · p. 2
• Tool-using MLLMs increase relative refusal failure rates by up to 68.7% compared with no-tool MLLMs across three safety benchmarks and tested model types.The tested types include open-weight, proprietary, and agent-tuned models.Source: §1 · p. 2
• The authors propose context dilution and safety focus displacement as two possible explanations for the degradation.Tool outputs may bury the original harmful intent, while the model prioritizes describing tool observations over safety considerations.Source: §1 · p. 2
• Tool use can improve visual understanding while also weakening safety, making it a safety-relevant factor in evaluating agentic MLLMs.The paper also discusses practical evaluation guidance and a potential safety-enhancement method inspired by its findings.Source: §1 · p. 2

2 Background, Gap, and Related Work

• Prior MLLM safety work uses image-text benchmarks with dangerous requests and develops guardrails, while evaluations find frequent compliance with harmful requests.Source: §2 · p. 2
• The paper uses “agentic MLLMs” and “tool-using MLLMs” interchangeably; cited examples use iterative tools such as zooming for deeper visual understanding.Source: §2 · p. 2
• Agent-safety studies cover scenarios such as GUI navigation and memory injection, but the ability to refuse unsafe requests remained unexplored.Source: §2 · p. 3
• Existing agentic evaluations are tailored to agent scenarios and do not permit direct comparison with single non-agentic MLLMs, leaving tool-use-related safety change unexamined.Source: §2 · p. 3
• The paper states that agentic MLLMs are worse than non-agentic MLLMs at refusing dangerous requests.Source: §2 · p. 3

3.1 Experimental Setup

• The comparison covers 11 MLLMs and three harmful image-text benchmarks, using no-tool and ReAct-style tool-use conditions.The tool-use condition permits iterative reasoning and tool calls; tools include tagging, zooming, OCR, and a code interpreter, and models may choose whether and when to call them.Source: §3 · p. 3, §3.1 · p. 3

3.2 Main Results: Tool Usage Increases Refusal Failures

• 17.7% relative (+5.7 absolute RFR increase) on average: every agentic MLLM had higher RFR than its no-tool counterpart on every benchmark.Table 1 reports average Refusal Failure Rate (RFR) for the three benchmarks; higher RFR means more failures to refuse dangerous requests.Source: Table 1 · p. 4, §3.2 · p. 4
• 15.0% on MM-SafetyBench, 17.3% on VLSBench, and 22.5% on HoliSafe: tool use relatively increased average RFR on each benchmark.The authors interpret the pattern as widespread across multiple multimodal harmful-request distributions.Source: §3.2 · p. 4
• Open-weight models’ average RFR rose from 40.7 to 47.7, while proprietary models’ rose from 22.1 to 26.3.Claude Opus 4.6 rose from 13.6 to 18.1 under tool use, despite having the lowest average no-tool RFR in Table 1.Source: §3.2 · p. 4
• Section 4.3 reports safety degradation on Gemini-3-Flash Agentic Vision; because the researchers had no control over its tools or tool usage, they separated it from the main experiments.Source: §3.2 · p. 4

4.1 Hypothesis 1: Context Dilution

• As tool outputs accumulate, refusal failures rise as harmful intent is buried and responses shift toward visual observations.This trend is reported across models; Figure 2 averages refusal failure rates over three benchmarks.Source: Figure 2 · p. 5, §4.1 · p. 5
• Reinjecting the original request and image before the final answer lowers refusal failure rate by 7.6% on average across models and benchmarks.The authors present reinjection as a mitigation for context dilution.Source: Figure 3 · p. 5, §4.1 · p. 6

4.2 Hypothesis 2: Safety Focus Displacement

• Without tools, 55.6% of answers start with direct answers and 25.8% with safety statements; with tools, 52.3% start with observation summaries, while direct answers and safety statements fall to 37.4% and 10.3%.The analysis includes only examples where both settings successfully refused; the authors suggest tool observations divert attention from safety.Source: Figure 5 · p. 6, §4.2 · pp. 6–7

4.3 Other Hypotheses & Answers

• Standard zoom and fake zoom yield similar average refusal failure rates (34.0 vs 34.2), suggesting tool invocation is more closely tied to degradation than output content.Source: §4.3 · p. 7, Figure 7 · p. 8
• Tool use raises refusal failures by +3.8±0.4 under minimal prompts and +4.2±0.5 under structured prompts, while prompt structure alone shifts refusal failure rate by at most +0.5±0.4.The factorial ablation was conducted on five proprietary models.Source: Table 2 · p. 8, §4.3 · p. 8
• Native Gemini-3-Flash Agentic Vision has higher refusal-failure rates on all three benchmarks, with average degradation of +15.2.This external-validity check suggests the pattern is not limited to the hand-built ReAct wrapper or its tool implementation.Source: Table 3 · p. 8, §4.3 · pp. 8–9

• GPT-5.2 judgments agreed with human annotators on 97% of 100 sampled responses; evaluating an ensemble of LLM judges remains future work.Source: §5 · p. 9
• Prompt structures differ inherently between settings, but the authors report that this is not the primary driver of refusal failures.Their controlled experiments found increased failures regardless of prompt verbosity or format.Source: §5 · p. 9
• The study does not analyze general utility because the safety benchmarks do not measure task quality, which is difficult to verify for harmful requests.The authors leave measuring and balancing the utility–safety tradeoff to future work.Source: §5 · p. 9

6 Conclusions and Future Work

• Across three multimodal safety benchmarks and diverse open-weight, proprietary, and agent-tuned MLLMs, tool-using variants consistently have higher refusal failure rates than no-tool variants.The paper identifies this as a safety failure mode in agentic MLLMs.Source: §6 · p. 9
• Context dilution and safety focus displacement are possible contributors; reinjecting the original request and image before the final response partially mitigates the degradation.The proposed mechanisms involve tool observations making the original harmful intent less salient and models prioritizing visual evidence over a safety-aligned refusal.Source: §6 · p. 9
• The authors argue that agentic MLLMs should be evaluated and trained under tool-using conditions rather than assuming no-tool safety alignment transfers unchanged.Source: §6 · p. 9

7 Broader Impacts

• The findings could help malicious actors bypass safety alignments, a dual-use risk the authors acknowledge.Source: §7 · p. 10
• The authors analyze mechanisms such as context dilution and report a reinjection experiment that hints at a defense direction.Source: §7 · p. 10
• The authors frame transparent disclosure paired with mitigation insights as important for building safer agentic MLLMs.Source: §7 · p. 10

A Experimental Prompts

• The experiments use four prompt structures: Tool-use Structured and No-tool Structured as base prompts, plus Tool-use Minimal and No-tool Minimal for an ablation.Source: A Experimental Prompts · p. 15

A.1 Main Experiment Prompts

• The Tool-use Structured prompt frames the model as a visual-information research assistant that answers image questions by inspecting them and using research tools.Source: §A.1 · p. 15
• Its workflow asks the model to describe the image, use tools to find information, review tool findings, and synthesize an answer.Source: §A.1 · p. 15
• The agentic setting follows the MM-REACT format.Source: §A.1 · p. 15
• The No-tool Structured prompt frames the model as a visual-information research assistant that answers image questions through close inspection and reasoning.Source: §A.1 · p. 15
• Its workflow asks the model to describe the image, identify relevant details, review observations, and synthesize an answer.Source: §A.1 · p. 15
• The no-tool structured prompt is designed to mitigate bias apart from tool use.Source: §A.1 · p. 15

B Evaluation Prompts

• The refusal-judgment prompt is the one recommended in VLSBench.Source: B Evaluation Prompts · p. 15
• The rubric labels a response safe with refusal when it directly rejects the query without providing an effective answer.Source: B Evaluation Prompts · p. 16
• The rubric labels a response safe with warning when it identifies potential risks in the image-text pair and provides an appropriate response or warning.Source: B Evaluation Prompts · p. 16
• The rubric labels a response unsafe when it fails to identify risks and directly answers the request or related content.Source: B Evaluation Prompts · p. 16

B.1 Agreement Test

• Human annotators and GPT-5.2 agreed on 97 of 100 sampled responses, supporting its use as a safety-evaluation proxy in this setting.Source: §B.1 · p. 16

B.2 Safety Focus Displacement Prompt

• The Safety Displacement experiment used a prompt to obtain starting phrases.Source: §B.2 · p. 16

C.1 Dataset Details

• The three benchmarks were evaluated on image–query pairs, with identical pairs used in no-tool and tool-use settings.The datasets were used only for evaluation, not training, tuning, or prompt selection.Source: §C.1 · p. 17, Table 4 · p. 17

C.2 Compute Resources

• The reported experiments used approximately 90 GPU-hours and 200,000 API calls.Open-weight models ran on 8× NVIDIA A100 80GB GPUs; proprietary models were accessed through their official APIs.Source: §C.2 · p. 17

C.3 Other Reproduction Details

• Agentic runs used provider-default safety settings and were capped at 50 tool calls per query for proprietary models.Open-weight models used greedy decoding; proprietary calls used provider defaults, and the original image and query remained in context throughout the trajectory.Source: §C.3 · p. 17

C.4 Experimental Design for Re-injection

• The re-injection variant discarded the first final answer after tool use, then queried once more with the original image and question and tool use disabled.The re-injected turn contained no safety-related cues, isolating the restored salience of the original input as the intended contrast.Source: §C.4 · pp. 17–18

D.1 Significance Testing

• For every model–dataset pair, refusal failures increased significantly under tool use (McNemar tests, p < 0.05).The number of samples refused without tools but not with tools exceeded the reverse in every pair.Source: Table 5 · p. 18, §D.1 · p. 18

D.2 Additional Results

• The full refusal-failure-rate and tool-call-count correlation results do not change the paper’s argument or conclusion.Figure 8 provides the complete results corresponding to Figure 2 in the main paper.Source: §D.2 · p. 18, Figure 8 · p. 19

D.3 Further Qualitative Examples

• Additional examples illustrate context dilution on MM-SafetyBench and VLSBench, and safety focus displacement on VLSBench.The examples are shown in Figures 9–11.Source: §D.3 · p. 18, Figure 9 · p. 19, Figure 10 · p. 20