SAFIRE: Safety-Critical Benchmark for Fine-grained Fire and Smoke Understanding in Multimodal LLMs

• The listed authors are Pengfei Li, Naufal Suryanto, Sicheng Zhang, Mohammad Alsharid, and Muzammal Naseer.Source: SAFIRE · p. 1
• The SAFIRE project page is https://risys-lab.github.io/SAFIRE/.Source: SAFIRE · p. 1

• Existing fire-smoke studies remain limited to narrow recognition tasks, small privately curated datasets, and minimal contextual annotation (Han et al., 2024b).Source: §1 · p. 1
• Firefighting water vapor, warm lighting, and controlled pyrotechnics can be mistaken for smoke or fire when broader scene cues are ignored.Source: §1 · p. 2
• SAFIRE combines 83K captioned images across 20 scenarios with 193K MCVQA pairs from a 9.7K-image subset, spanning ten evaluation dimensions.Source: §1 · p. 2
• 61.9% average MCVQA accuracy across ten open-source MLLMs highlights remaining gaps in safety-critical reasoning.Source: §1 · p. 2

• BowFire (Chino et al., 2015) and F. Yuan used small binary setups, while DFAN introduced scene-specific categories.Source: §2 · p. 2
• Despite dataset advances, video-derived frames and narrow scenario coverage restrict contextual diversity; the comparison identifies SAFIRE by its scale, scenario diversity, and wide coverage.Source: §2 · pp. 2–3

• Prior resources include localization, paired desmoking images, segmentation, and sensor-augmented datasets, extending beyond basic classification.Source: SAFIRE (Ours) · p. 3
• Among 23 representative datasets, classification dominates (17/23), only 2/23 include image captions for reasoning, and none include VQA pairs.Source: SAFIRE (Ours) · p. 3
• Prior benchmarks seldom systematically evaluate fire-smoke reasoning; SAFIRE combines diverse scenarios, image-caption pairs, and MCVQA annotations to assess perception, reasoning, and alignment.Source: SAFIRE (Ours) · p. 3

3 SAFIRE Benchmark

• SAFIRE contains 83K captioned images across 20 real-world scenarios and 193K MCVQA pairs from a curated 9.7K-image subset.The benchmark also includes complementary image-caption annotations for perceptual understanding and higher-level reasoning.Source: §3 · p. 3

3.1 Data Collection and Structuring.

• Multi-source images are filtered, then 9.7K samples are structured into ten-dimension MCVQA and five-group, twenty-label scene subsets.Annotation generation uses structured prompts; GPT-5.4 verification, MLLM majority voting, and manual review support quality control.Source: §3.1 · p. 4

3.2 MLLM Evaluation.

• MLLM evaluation uses image, scenario label, and four-option questions, measuring accuracy by scenario and reasoning dimension.Models are prompted to respond with one option letter; tested skills include spatial awareness, counting, intensity estimation, and semantic classification.Source: §3.2 · p. 5

3.3 Vision-Language Encoder Evaluation.

• Vision-language encoders are evaluated with zero-shot and few-shot protocols for fire-smoke scene understanding.The zero-shot protocol selects the class with the highest image-text cosine similarity; the linear SVM probe trains on frozen image embeddings with 1%–30% supervision.Source: §3.3 · p. 5

4.1 Experimental Setup

• Experiments use a curated 9.7K-image test subset and evaluate both MLLMs and vision-language encoders.MLLM inference uses an NVIDIA A100 multi-GPU cluster with vLLM, automatic device mapping, and bfloat16 precision.Source: §4.1 · p. 5

4.2 MLLM Evaluation Results

• 61.87% average accuracy across the twenty-scenario MCVQA benchmark shows substantial room for improvement in safety-critical multimodal reasoning.InternVL 38B exceeds its 8B counterpart by around four points, while gains diminish beyond a certain scale.Source: §4.2 · p. 6, Table 3 · p. 6
• SAFIRE performance varies by reasoning dimension: Classification and Human Presence reach 76.4% and 90.3% accuracy, while Emotional Response, Spatial Correlation, and General Reasoning score 61.7%, 42.9%, and 53.3%, respectively.Position Identification averages 57.6%.Source: §4.2 · p. 6
• Reasoning-enhanced models reach around 62–66% overall accuracy, with a 2.7-point decline for the 9B model and a 0.3-point gain for the 32B series.The results describe explicit reasoning gains as modest and sometimes negative.Source: Table 3 · p. 6, §4.2 · p. 7
• The manually audited 2,800-MCVQA subset preserves broad model trends, while scenario-prior ablation indicates results are mainly driven by image-grounded reasoning.The subset is not intended to isolate label correction’s causal effect; the authors report that residual annotation noise does not change the main findings.Source: §4.2 · p. 7

4.3 Multimodal Encoder Evaluation Results

• Zero-shot encoders reach up to 72.2% accuracy, but results are weaker for visually cluttered or ambiguous scenes such as Meteor and Waste Disposal Scene.Smoking, Flare Stack, and Volcano exceed 80% accuracy; Firework accuracy ranges from over 90% in MetaCLIP to 71% in CLIP_ViT_L14.Source: §4.3 · p. 8, Figure 6 · p. 8
• EvaCLIP leads zero-shot models on F1=0.70, Recall@1=0.71, and Precision=0.71, despite MetaCLIP having the highest overall accuracy.MetaCLIP’s macro-level metrics are consistently lower than EvaCLIP’s.Source: §4.3 · p. 8, Figure 7 · p. 8
• With 30% supervision, CLIP+SVM achieves F1=0.79 and Precision=0.83, while 7% supervision yields F1=0.63.Performance improves consistently as label supervision increases.Source: §4.3 · p. 8, Figure 7 · p. 8

5 Conclusion

• SAFIRE comprises over 83K captioned images across 20 scenarios and 193K MCVQA pairs, supporting ten open-source MLLMs and encoder evaluations.Ten open-source MLLMs average 61.9% MCVQA accuracy; CLIP-style encoders reach 72.2% in zero-shot classification, while 7% supervised data boosts fire-scene classification accuracy from 20.1% to 64.5%.Source: §5 · p. 9

Limitations

• SAFIRE’s retrieval and alignment evaluations use limited natural-language prompts; broader syntactic and semantic variations, tuning, or renaming are proposed as future directions.The authors suggest these may further improve generalization and text–image alignment in safety-critical scenarios.Source: Limitations · p. 9
• The evaluated models use general-domain pretraining, and adaptation tests cover only a linear probe and one supervised fine-tuning configuration.Evaluation across more architectures and adaptation methods remains future work.Source: Limitations · p. 9
• Most annotations are model-generated and refined through GPT-5.4 and small-model voting, with human checks on flagged samples; biases may persist from the generator’s capabilities at design time.The authors call for ongoing audits and community-driven validation to ensure fairness.Source: Limitations · p. 9

A Ethical Statement

• The benchmark’s data and pretrained models were obtained or used under the stated public-source, author-experiment, and open-source-license conditions.The authors say AI tools assisted with data cleaning and language clarity, while the authors verified the conceptual and scientific content.Source: A Ethical Statement · p. 12

B.1 Dataset Collection

• Image collection uses multi-source web searches and filtering, while raw provenance counts are approximate candidates reduced to the final 83K images after filtering and inspection.The benchmark is released for non-commercial research use; the controlled experiment protocol received university ethics approval.Source: §B.1 · p. 12, §B.2 · p. 12

B.3 Dataset Cleaning

• Candidate scenarios were assessed for image quality and noise and for semantic relevance; scenarios with high false-positive rates or conceptual redundancy were discarded.Examples include Satellite Massive Fire, which often confused fire with laminar flow, haze, or cloud cover, and categories that overlapped with others.Source: §B.3 · p. 12
• 61.2% of the initial data was retained, leaving 83K high-quality images after reduction from 135.4K auto-collected images to 91K after semantic examination and then manual inspection.The authors describe the filtering as producing a cleaner, more diverse, and semantically coherent dataset.Source: §B.3 · pp. 12–13

C.1 Annotation Structure and Benchmark Objectives

• The 83K image-caption pairs capture semantic attributes and safety cues, while 193K questions on a 9.7K-image subset assess ten reasoning dimensions.The two branches support encoder evaluation and fine-grained MLLM reasoning, respectively.Source: §C.1 · p. 13

C.2 Prompting Strategy and Instruction Design

• Standardized prompts built on Qwen2.5-VL-72B-Instruct generate annotations across ten reasoning dimensions, with instructions to follow image content and allow “None of the above” when options do not fit.The instructions also specify context awareness, visual description, and structured JSON output.Source: §C.2 · pp. 13–14

C.3 MCVQA Design and Reasoning Dimensions

• Each selected MCVQA-active image receives approximately 20 questions, ranging from scene classification and spatial recognition to context, activity, and semantic ambiguity reasoning.Questions cover scenario type, fire position and counts, smoke and flame attributes, human presence, trends, and emergency response.Source: §C.3 · p. 14

C.4 Structured Output Format

• Annotations use a unified JSON schema containing image ID, caption, key indicators, and MCVQA entries to support downstream compatibility and reproducibility.Figure 8 depicts the majority-voting pipeline for cleaning multi-scenario MCVQA answers.Source: §C.4 · p. 14, Figure 8 · p. 14

C.5 Handling MCVQA Failures

• When generation returns captions without MCVQA pairs, the captions are retained for the CLIP branch, preserving aligned image-text examples.Complete failures return no output; partial failures contain captions but no MCVQA pairs.Source: §C.5 · p. 14

C.6 GPT-5.4 Verification and Majority Voting for MCVQA Correction

• Verification combines scenario-specific prompting, GPT-5.4 consistency checks, and three independent non-Qwen verifiers, with flagged disagreements prioritized for human review.The authors report that verifier agreement with original answers does not rely on one dominant model, and voting can flag ambiguous questions.Source: §C.6 · pp. 14–15
• Three verifier models produced a clear majority for 97.09% of MCVQA samples, while 7,137 samples (3.70%) were double-reviewed through the voting process.Voting flagged disagreement or conflicts with the original answer for manual review rather than automatically relabeling data; the table reports inter-model consistency.Source: Figure 8 · p. 14, §C.6 · p. 15, Table 7 · p. 15

D Annotation Error Analysis

• In the 2,800-sample audit, the remaining annotation error rate was 12.9%, with most errors involving visual-grounding ambiguities, semantic mismatches, or answer-format inconsistencies.On the manually corrected subset, model accuracies were 1.30 to 5.36 percentage points higher than on the full benchmark, but relative model ordering remained broadly consistent; the subset is a robustness check, not an isolated estimate of label correction’s effect.Source: Table 6 · p. 15, D Annotation Error Analysis · p. 16, Table 9 · p. 16

E Dimension Correlation Analysis

• The strongest reported dimension association was Classification–People Emotional Response at 0.410, while Human Presence associations with Position Identification, Fire–smoke Spatial Correlation, and Target Counting were 0.030, 0.040, and 0.060.The authors characterize even the strongest association as moderate and conclude the dimensions remain sufficiently distinct for multidimensional evaluation.Source: E Dimension Correlation Analysis · pp. 16–17, Table 10 · p. 16

F Scenario-Prior Ablation Study

• With images absent, adding the scenario name raised accuracy from 29.26% to 39.17%; adding the image to question-only input raised it from 29.26% to 64.13%.Removing scenario information from the original setting reduced accuracy by 0.76 percentage points, from 64.89% to 64.13%; the authors conclude the main protocol is primarily image-grounded.Source: F Scenario-Prior Ablation Study · p. 17, Table 11 · p. 17

G Evaluation Pipeline Algorithms

• The MCVQA evaluation loads an image with its scenario label and question, extracts the model’s final answer, and reports accuracy by scenario and dimension.Ambiguous or missing final answers are invalid; CLIP classification separately uses cosine similarity for zero-shot predictions and linear SVMs trained on 1%–30% of labeled image features.Source: G Evaluation Pipeline Algorithms · p. 17

H Model Configuration

• The evaluated MLLMs include Qwen, GLM, Gemma, InternVL, and other model families, while encoder experiments use CLIP, MetaCLIP2, EVACLIP, and OpenCLIP on a 9.7K-image test subset.Models were obtained from official repositories and evaluated with default inference settings for consistency and comparability.Source: H Model Configuration · pp. 17–19

I Bias Analysis on Generator-family Model and Human Performance

• 82.5% human accuracy on a sampled 700-question subset exceeds the best Qwen-family result of 65.8%, while Qwen and non-Qwen averages are 64.6% and 60.6%.Human performance is a reference estimate, not a paired full-set comparison; model families were evaluated under the same protocol.Source: Table 12 · p. 19, I Bias Analysis on Generator-family… · p. 19

J MLLM Supervised Fine-Tuning Study

• 70.83% held-out accuracy from 63.02% (+7.81 percentage points) after Qwen3-VL-8B supervised fine-tuning on SAFIRE.All 20 scenarios and all 10 reasoning dimensions improved; scenario gains ranged from +5.07 to +10.26 percentage points, and reasoning-dimension gains from +1.91 to +16.29 percentage points.Source: Table 13 · p. 20, J MLLM Supervised Fine-Tuning Study · p. 20

K Visual Diversity Across Fire-Smoke Scenarios

• SAFIRE’s 20 scenarios span common and uncommon natural, industrial, civil, controlled, recreational, and symbolic contexts, with examples capturing within- and between-scenario variation.The examples include large-scale fires, culturally embedded and industrial events, and smaller or ambiguous sources.Source: K Visual Diversity Across Fire-Smoke… · p. 20, Figure 10 · p. 21

L A Full Generated MCVQA Example

• The Volcano example’s 20 MCVQA questions cover visual properties, likely cause, life presence, public reaction, idiom interpretation, and plausible future developments.The example is presented as illustrating factual visual grounding and inferential reasoning in the benchmark.Source: L A Full Generated MCVQA Example · p. 20, Figure 14 · p. 24, Figure 15 · p. 25