Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models

• Figure 1 presents safety issues of OLLMs.Source: Figure 1 · p. 2
• The figure focuses on safety concerns in omni-modal language models.Source: Figure 1 · p. 2
• Its scope is the safety evaluation of OLLMs.Source: Figure 1 · p. 2

Motivation

• Existing safety benchmarks lack audio-visual joint samples, broad modality coverage, and parallel cases for evaluating cross-modal safety consistency.Source: §1 · p. 2

Metrics

• The benchmark evaluates safety only on understood inputs using C-ASR and C-RR, then measures cross-modal consistency with CMSC-score.This separates comprehension failures from safety failures and compares performance across 24 parallel subcategories.Source: §1 · p. 2

Results

• Safety weakens sharply as modality combinations become more complex, with audio-visual inputs most effectively triggering vulnerabilities in most OLLMs.All eleven models show the pattern omni-modal inputs < dual-modal inputs < unimodal inputs in safety scores.Source: §1 · p. 2

Safety Alignment

• Existing alignment methods are less effective at inference time than post-training with image-text datasets, while modality-specific training creates major out-of-distribution issues.Adding more modalities does not fully resolve the challenge because complex combinations require harder safety reasoning and make equal in-distribution data less effective.Source: §1 · p. 2

Benchmark

• Omni-SafetyBench provides 23,328 safety test instances across 24 modality variations derived from 972 seed samples through modality conversion.It covers unimodal, dual-modal, and omni-modal paradigms, including audio-visual joint harm cases.Source: §1 · p. 2

2.1 Dataset Composition

• Omni-SafetyBench spans unimodal, dual-modal, and omni-modal paradigms across 24 modality variations derived from a unified seed dataset.The omni-modal paradigm redundantly integrates audio, visual, and text modalities for harmful instructions.Source: §2.1 · p. 3, Table 2 · p. 4
• 972 harmful samples from eight categories were manually converted into textual seed instances and transformed into multiple modalities.The transformations include diffusion-generated images and videos, typographic visuals, hybrid visuals, and text-to-speech audio.Source: §2.2.1 · p. 3, §2.2.2 · p. 3

2.2 Generation Quality Verification

• Semantic consistency across modality transformations is enforced through a two-tier verification pipeline using model agreement and manual review for disagreements.Diffusion-based content is checked by Gemini-2.5-Pro and GPT-4o, while failed samples can be re-captioned and regenerated.Source: §2.2.3 · p. 3, Table 5 · p. 4
• Noise-level adjustment increased the reported verification rate to 100% while balancing audio intelligibility against the intended perturbation.The passage reports this result for the relevant generation process but does not specify the initial rate.Source: §2.2.3 · p. 4
• Final samples were composed for each modality variation with modality-specific instruction edits to preserve logical consistency between inputs and descriptions.For example, descriptions were changed to refer jointly to image and audio when both modalities were present.Source: §2.2.4 · p. 4

2.3 Evaluation

• The framework first evaluates safety on single modality variations and then measures cross-modal safety consistency across all tested variations.Table 8 reports average C-ASR, C-RR, Safety-score, and overall CMSC-score by modality paradigm and across the benchmark.Source: §2.3 · p. 4, Table 8 · p. 5

2.3.1 Single Modality Variation Evaluation

• Each response receives comprehend, harmful, and refuse flags determined by majority vote among five LLM judges, with 3-2 splits marked uncertain.Uncertain cases are routed to manual review, and conditional metrics exclude failures caused by incomprehension.Source: §2.3.1 · pp. 4–5
• C-ASR measures harmful-content probability given comprehension, while C-RR measures refusal probability under the same condition.The comprehension probe uses a separate non-harmful descriptive question to assess whether the model understood the input.Source: §2.3.1 · p. 5
• Safety-score combines C-ASR and C-RR with λ=0.5, penalizing harmful content while rewarding refusal and producing values from 0 to 1.Higher C-ASR lowers the score, whereas higher C-RR raises it; harmful content receives priority in the safety check.Source: §2.3.1 · p. 5

2.3.2 Cross-Modal Safety Consistency

• CMSC-score converts the standard deviation of modality-specific Safety-scores into e^(-ασ), with α=5 and values closer to 1 indicating greater consistency.The metric measures dispersion across N subcategories rather than average safety alone.Source: §2.3.2 · p. 5

Results and Analysis

• Only three of eleven models exceed 0.6 in both average Safety-score and overall CMSC-score: Qwen3-Omni-30BA3B, gemini-3-pro, and gemini-2.5-pro.gemini-2.5-pro performs best at approximately 0.8 on both dimensions, while Qwen3-Omni scores approximately 10 points lower across both dimensions.Source: §3.2.1 · p. 6
• Safety declines consistently as modality complexity increases, with omni-modal inputs scoring below dual-modal and unimodal inputs across all eleven OLLMs.Audio-visual joint inputs are most effective at triggering vulnerabilities in most models, and seven of eleven models are most vulnerable to omni-modal inputs.Source: §3.2.2 · p. 6, Figure 4 · p. 6
• Eight of eleven models fall below a 0.50 Safety-score in their worst modality case, while the weakest model reaches only 0.19.The strongest model’s worst-case score drops from 0.78 to 0.60, and larger worst-case gaps align with lower cross-modal consistency.Source: §3.2.3 · p. 6, Table 9
• The benchmark classifies OLLMs into Robustly Safe, Selectively Risky, Consistently Risky, and Critically Unstable groups using average Safety-score and overall CMSC-score.Even the Robustly Safe models achieve only around 0.8 in both dimensions, leaving measurable room for improvement.Source: §3.2.4 · p. 6

Reliability Analysis of LLM-Based Judgment

• The evaluation uses majority voting by five LLM judges over comprehension, harmfulness, and refusal, routing 3-2 splits to manual review.The judges achieve 89.1% average pairwise agreement and Fleiss’ κ=0.76; LLM majority voting reaches 92.2% agreement with human consensus on 500 samples.Source: §3.3 · p. 7, §3.3.1 · p. 7, §3.3.2 · p. 7

Effectiveness of Inference-Time Safety Alignment on OLLMs

• Inference-time methods leave C-ASR above 30% and Safety-score below 0.55 for dual-modal and omni-modal cases, underperforming post-training approaches.The evaluated methods are ECSO, CoCA, and ShiftDC on Minicpm-o-2.6, with performance measured using Omni-SafetyBench and general capability using OmniBench.Source: §4.1 · p. 7, Table 13 · p. 8
• Inference-time methods cannot fundamentally change a model’s safety understanding and provide only temporary adjustments, especially for models with major safety flaws or omni-modal inputs.This limitation explains their weaker performance in complex scenarios rather than establishing a durable safety improvement.Source: §4.1 · p. 7

Effectiveness of Post-Training Safety Alignment on OLLMs

• Post-training alignment shows a diagonal effect: performance gains peak when evaluation modalities match the training data and drop substantially on other modalities.The study compares two text-only datasets and two image-text datasets using PGR across Omni-SafetyBench modality settings.Source: §4.2 · p. 8, Table 13 · p. 8
• Modality-augmented training slightly improves in-distribution safety over original-source training, but PGR is lowest for omni-input settings.Equivalent amounts of modality-matching training data are less effective for omni-input harmful cases, which remain inherently challenging.Source: §4.2 · p. 8, Figure 5 · p. 8

Benchmarks for Safety Evaluation

• Existing benchmarks separately target text, vision, audio, or video models but do not comprehensively evaluate safety across diverse OLLM modality combinations.The cited benchmark families include SafetyBench and SALAD-Bench for text, MM-SafetyBench and FigStep for vision, and AudioTrust, VA-SafetyBench, and Video-SafetyBench for audio or video.Source: §5 · p. 8

Multimodal Safety Alignment Methods

• Adding modalities can compromise safety, leaving vision-language and audio-language models vulnerable to non-text interventions such as image-to-caption conversion or contrastive decoding.These interventions are discussed as existing multimodal safety-alignment approaches.Source: §5 · pp. 8–9

Conclusion

• Omni-SafetyBench is presented as the first comprehensive benchmark for OLLM safety evaluation, using Safety-score and CMSC-score to expose key vulnerabilities.Evaluation of eleven OLLMs finds only three models above 0.6 in both metrics and identifies major challenges in existing alignment algorithms.Source: §6 · p. 9

• Comprehension failures can artificially inflate safety metrics when models misunderstand harmful multimodal queries and therefore produce non-harmful outputs.Judge models may rate these responses as safe even though the models did not correctly understand the input intent.Source: A Case Studies of Unfair Safety… · p. 10

• Evaluation prompts assess both whether the model produces harmful output and whether its answer indicates refusal.Source: B Evaluation Prompts · p. 10

• Minicpm-o-2.6 refuses the harmful query in the unimodal setting but generates detailed unsafe steps in the omni-modal setting.The dual-modal response avoids explicit harmful content but does not register as a refusal.Source: C Attack Cases · p. 10
• Refusal assessment is performed separately from harmful-output assessment using dedicated evaluation prompts.Source: C Attack Cases · p. 12, Figure 8 · p. 12
• The same harmful seed data produces divergent response behavior when transformed across unimodal, dual-modal, and omni-modal inputs.Figure 9 marks dangerous or potentially dangerous content in red and safe content in green.Source: C Attack Cases · p. 10, Figure 9 · p. 13