PaperlayerView exact revision on arXiv

Source-linked AI summary

Distilling Vision-Language Models for On-Device Fire Understanding

Mohammad Kazzazi, Zixuan Liu, Siavash Khajavi

arXiv:2609.05782v1cs.AI

TL;DR

Conventional fire sensors lack semantic context, while large VLMs are difficult to deploy on embedded hardware. This paper distills fire-specialized VLM teachers into lightweight students and evaluates them on-device, finding that compact models retain most fire-understanding capability, with Qwen2.5-0.5B offering the strongest overall deployment trade-off.

  • Problem

    Large VLMs can provide semantic fire understanding but cannot readily satisfy the latency, privacy, and embedded-resource constraints of fully on-device fire detection.

  • Method

    The paper fine-tunes large VLMs for fire understanding and distills their behavior into lightweight students using teacher imitation and ground-truth supervision.

  • Results

    Compact students retain most of their teachers’ fire-understanding capability, and Qwen2.5-0.5B provides the strongest overall deployment trade-off across tested families and scales.

  • Takeaways & Limitations

    On-device deployment must be evaluated jointly across reasoning accuracy, safety-relevant errors, latency, memory, and model failure modes.

  • Takeaways & Limitations

    Evaluating distillation only on positive fire-scene reasoning can hide severe failure on negative examples, so validation must include non-fire scenes.

Abstract

from arXiv · show

Vision-language models (VLMs) offer a promising alternative to conventional fire detection systems by reasoning about the semantic context of a scene and thus reducing false alarms, yet their large model size makes deployment on embedded fire sensors impractical. In this paper, we study how domain-specialized VLMs can be compressed for fully on-device deployment without losing the safety-critical behavior required for fire detection. We develop a teacher-student knowledge distillation framework in which large VLMs fine-tuned for fire understanding can be distilled into lightweight students. Experiments across multiple VLM families and model scales show that compact students preserve most of their teachers' fire-understanding capability. We further deploy the distilled models on our commercial Detectium fire detection sensor and jointly evaluate reasoning accuracy, latency, and memory usage. The results show that compression and deployment affect not only accuracy but also model failure modes, with Qwen2.5-0.5B providing the strongest overall deployment trade-off. Our findings provide broader guidance for deploying domain-specialized VLMs in resource-constrained, safety-critical settings.

1 Introduction

Conventional fire sensors often confuse benign high-temperature or flame scenarios with hazardous fires, while VLMs can use scene context to reduce false alarms. This paper develops and evaluates distilled, on-device fire-understanding models to address the conflict between semantic reasoning and embedded deployment constraints.

  • Traditional smoke and thermal sensors rely on low-level signals, causing false alarms for benign flames and controlled high-temperature processes.False alarms require human verification, waste resources, and can erode trust.
  • VLMs can distinguish controlled from uncontrolled flames and assess whether an observed fire poses a genuine safety risk.
  • Large VLMs cannot run on embedded fire sensors, while cloud inference conflicts with the seconds-level latency required for early fire detection.
  • The paper fine-tunes Qwen, Llama, and Gemma teachers, then distills their fire-domain knowledge into lightweight students across multiple parameter scales.
  • Compact students retain most teacher capability, but increasing student size yields limited accuracy gains and can shift deployment failure modes.
  • Qwen2.5-0.5B provides the strongest overall trade-off across fire understanding, detection errors, latency, and memory usage.

2 Related Work

Fire-detection research is moving from localization toward semantic risk understanding with VLMs, but embedded deployment remains insufficiently studied. Knowledge distillation offers an established route to smaller deployable VLMs.

  • VLMs for Fire Detection: VLMs describe fire scenes and assess risk, while SmokeBench and DetectiumFire evaluate burning objects, environments, and risk levels beyond binary detection.
  • VLMs for Fire Detection: Existing fire-domain VLM studies primarily evaluate offline or assume cloud-scale inference, leaving embedded-sensor deployment unexplored.
  • Knowledge Distillation: Knowledge distillation transfers large-teacher capability into smaller students and has become a standard route to deployable VLMs.

3 Knowledge Distillation for Fire Understanding

The paper builds fire-specialized teachers through staged visual-language training, then distills their behavior into initialized lightweight students. The student objective combines teacher imitation with ground-truth supervision to avoid a fire-prediction failure mode.

  • The framework first fine-tunes large VLMs for contextual fire understanding, then distills that fire-domain knowledge into lightweight students.
  • Teacher Model: Teacher construction begins by freezing the language model and vision encoder while training the vision-language adaptor on general image-caption data.This establishes general visual-language alignment before fire-specific supervision.
  • Teacher Model: Task-specific fine-tuning uses fire-specific image-caption data together with general vision-language instructions to preserve visual grounding on unfamiliar scenes.
  • Student Model: Students receive the same general alignment initialization before being trained to reproduce teacher output distributions on fire-understanding examples.
  • Student Model: Optimizing KL divergence alone can make students predict fire for nearly every image, so the method adds cross-entropy supervision from ground-truth responses.

4 Distillation Evaluation

Across model families and scales, distilled students retain most teacher fire-understanding capability, but distillation and architecture choices affect fine-grained accuracy and safety-relevant detection. The full distillation objective preserves balanced fire/non-fire performance, whereas KL-only training can cause severe false-alarm behavior.

  • Overall performance: Teachers achieve near-perfect fire/non-fire detection, while fine-grained burning-object, environment, and severity reasoning remains more challenging.The evaluation covers four fire-understanding tasks on DetectiumFire, with the latter three evaluated on fire images only.
  • Distillation across families: Qwen2.5-0.5B retains 95.2% of burning-object accuracy, 99.3% of environment and severity accuracy, and 99.96% of fire/non-fire accuracy versus its 7B teacher.The student uses roughly an order of magnitude fewer language-model parameters: 0.5B versus 7B.
  • Distillation across families: Student degradation is largest for burning-object identification, with typical teacher gaps of approximately 2.5-4.3, while fire/non-fire detection gaps are negligible.Environment and severity prediction fall between these extremes, with several students matching or exceeding their teachers.
  • Student scale: Scaling Qwen2.5 from 0.5B to 3B improves burning-object accuracy by only 1.37, while larger students provide no consistent fire/non-fire benefit.Qwen2.5 fire/non-fire accuracy decreases from 99.75% at 0.5B to 98.51% at 3B; latency and memory usage also grow with parameter count.
  • Vision encoder: Replacing CLIP ViT-L/336 with MobileCLIP-S2 causes larger accuracy losses than changing language-model size, including 7.83 in environment accuracy and 4.91 in severity for Qwen2.5-7B.Qwen3-8B loses 9.68 in burning-object accuracy, and this degradation is inherited by distilled students.
  • Distillation objective: Removing the cross-entropy term causes severe fire/non-fire specificity collapse, while adding it restores specificity to 97.52%-99.90% and recall to 99.43%-99.82%.The collapse is not mitigated by increasing capacity: Qwen3-4B reaches only 60.60% specificity without the cross-entropy term, and the failure emerges during distillation rather than from teachers.
  • Distillation objective: Positive-class fire-scene reasoning can conceal deployment failure because KL-only models are evaluated on fire images for those tasks and are not penalized for indiscriminate fire predictions.The ablation therefore supports validating domain-specific distillation on negative examples, not only positive-class reasoning performance.

5 On-device Deployment

The distilled students were evaluated on a CPU-only Detectium sensor for accuracy, safety-relevant errors, latency, and memory. Deployment largely preserved fire/non-fire accuracy, but compression shifted failure modes and made Qwen2.5-0.5B the strongest overall trade-off.

  • Deployment setup: The Detectium sensor has limited memory, no GPU or accelerator, and no network access during inference.All experiments therefore run entirely on the sensor CPU.
  • Accuracy after deployment: Exported students change fire/non-fire accuracy by at most 0.89 percentage points, while fine-grained reasoning can degrade more substantially.Llama-3.2-1B loses 4.07 in burning-object accuracy and 3.14 in environment accuracy after export.
  • Safety-relevant failure modes: Quantized CLIP-based students shift toward more false alarms, whereas MobileCLIP-based students are more likely to miss fires.Missed fires are more consequential than false alarms in practical fire detection, making the MobileCLIP failure mode undesirable despite its efficiency.
  • Latency and memory: TTFT accounts for approximately 83%-95% of total E2E latency across CLIP-based students.For Qwen2.5-0.5B, TTFT is 3818 ms and the remaining response generation takes 216 ms.
  • Latency and memory: MobileCLIP-S2 reduces TTFT substantially, but its unsupported export path means lower TTFT does not necessarily produce lower E2E latency.For Qwen2.5-3B, TTFT falls from 11598 ms to 1757 ms when replacing CLIP with MobileCLIP-S2.
  • Overall deployment trade-off: Qwen2.5-0.5B provides the strongest overall deployment trade-off at 4034 ms E2E latency, with the lowest memory usage and best overall post-export fire-understanding accuracy.Across evaluated Qwen models, E2E latency rises from 4034 ms to 16645 ms and RSS from 1880 MB to 9850 MB as model size increases.

B.1 Model Architecture

The models use the LLaVA-MoD architecture, combining a vision encoder, VL adaptor, and language-model backbone. Experiments vary both the vision encoder and compatible teacher-student language-model scales.

  • B.1 Model Architecture: Each model combines a vision encoder, a two-layer GELU VL adaptor, and an LLM that generates responses from projected visual features and instructions.The adaptor maps visual features into the language-model embedding space.
  • B.1 Model Architecture: CLIP ViT-L/336 produces 576 visual tokens from 336 × 336 images, whereas MobileCLIP-S2 produces 64 tokens from 256 × 256 images.The two encoders represent the paper’s accuracy–efficiency comparison.
  • B.1 Model Architecture: Teachers include Qwen2.5-7B, Qwen3-8B, Llama-3.1-8B, and Gemma-2-9B, with students distilled within compatible vocabularies.Student sizes range from Qwen2.5-0.5B to 3B, Qwen3-0.6B to 4B, Llama-3.2-1B to 3B, and Gemma-2-2B.
  • B.1 Model Architecture: Unlike LLaVA-MoD, the students are not sparsified and retain the standard architecture of their corresponding base language models.The Llama-3.1 and Llama-3.2 models share a tokenizer for token-level distillation.

B.2 Training Data

Training combines general vision-language alignment with fire-specific supervision and negative non-fire examples. Students then distill from the DetectiumFire training data using multiple task formulations, while an unseen prompt formulation tests generalization.

  • B.2 Training Data: Teachers also use filtered general vision-language examples to preserve visual-language capability during fire-specific specialization.The foundational stage uses the 558K-pair LLaVA-Pretrain dataset for visual-language feature alignment.
  • B.2 Training Data: Teacher task-specific fine-tuning uses DetectiumFire annotations covering burning objects, environments, severity, and non-fire images.The dataset includes 3,000 real-world non-fire images to teach rejection of benign scenes resembling fire.
  • B.2 Training Data: Student distillation uses only DetectiumFire data, pairing each fire image with four prompts and each non-fire image with three prompts.Each image-prompt pair is treated as a separate distillation example.
  • B.2 Training Data: The evaluation adds “Assess whether this is a fire incident or a non-fire scene,” a prompt formulation withheld during student distillation.This tests generalization to unseen question wording.

B.3 Training Configuration

Training uses separate teacher and student procedures with LoRA-based teacher fine-tuning and shared random seeds across student experiments. The reported configurations are organized in dedicated teacher-initialization and student-distillation hyperparameter tables.

  • B.3 Training Configuration: All models are trained on eight NVIDIA A100 GPUs with 80 GB of memory per GPU.
  • B.3 Training Configuration: Teacher models are fine-tuned with low-rank adaptation rather than full-parameter fine-tuning.Detailed teacher initialization and task-specific hyperparameters are reported in Table 6.
  • B.3 Training Configuration: Student initialization and distillation hyperparameters are reported separately from teacher training settings.Table 7 covers the student procedure.
  • B.3 Training Configuration: All experiments use the same three random seeds: 42, 123, and 456.

C.1 Ablation Study

The ablation shows that removing the cross-entropy term can preserve positive fire-scene reasoning while severely damaging fire/non-fire detection. This failure is therefore exposed by negative examples and is not inherited from the teachers.

  • C.1 Ablation Study: Removing LCE reduces fire/non-fire detection accuracy by 15.70 to 45.99 percentage points across most students.Qwen2.5-0.5B falls to 53.76% accuracy, while Qwen3-4B falls to 81.04%.
  • C.1 Ablation Study: Without LCE, fire-scene reasoning accuracy remains largely unchanged and sometimes improves because those tasks contain only fire images.Burning-object accuracy rises by 3.05 points for Llama-3.2-1B and 2.43 points for Qwen2.5-1.5B.
  • C.1 Ablation Study: Selecting the KL-only objective from positive-class reasoning alone can produce a model unsuitable for deployment.The ablation establishes that domain-specific distillation must be validated on negative examples.
  • C.1 Ablation Study: Teachers produce at most two false alarms and two missed fires, indicating that the student collapse emerges during distillation rather than from teacher behavior.
  • C.1 Ablation Study: Deployment evaluation uses a Raspberry Pi 5 with 8 GB memory, no GPU or NPU, a 7 GiB inference cap, and four CPU threads.

D.2 Export Pipeline

The export pipeline produces deployable CLIP-based students through LoRA merging, GGUF conversion, quantization, and CPU serving, while MobileCLIP-S2 students require a separate PyTorch path. Deployment measurements are constrained by hardware capacity and runtime differences.

  • CLIP-based export: CLIP-based students merge LoRA adapters, convert language models to GGUF, quantize them to Q8_0, and export the vision-language projector.The pipeline then serves the exported files with llama.cpp on the sensor CPU.
  • MobileCLIP-S2 export: MobileCLIP-S2 students cannot use the same GGUF pipeline because their FastViT backbone is unsupported, so they run directly in PyTorch using bfloat16 precision.Their LoRA adapters are applied at load time.
  • Comparability: MobileCLIP-S2 latency and accuracy are not strictly comparable with CLIP-based students because the configurations use different runtimes and numerical precision.They are reported primarily to characterize lightweight-encoder deployment behavior.
  • Evaluation setup: Inference requests use deterministic decoding with temperature 0, a maximum of 64 new tokens, and no image or prompt cache reuse.Each request independently re-encodes its unseen image, matching live-stream sensor processing.
  • Hardware constraints: Qwen2.5-3B with MobileCLIP-S2 and Qwen3-4B require approximately 8.8 GB and 9.9 GB of memory, exceeding the Detectium sensor capacity and preventing deployment.Their measurements are therefore reported for reference only.

E Additional Deployment Results

Export changes the safety behavior of distilled students, increasing false alarms for CLIP-based models while MobileCLIP-based models show a missed-fire failure mode. These deployment errors matter because missed fires carry greater safety consequences than alarms that humans can verify.

  • CLIP-based students: Every CLIP-based student produces more false alarms after export, reducing specificity and shifting predictions toward fire more frequently.Quantization therefore changes prediction behavior directionally rather than merely reducing overall accuracy.
  • Quantization sensitivity: 2.7 additional false alarms occur on average for CLIP-based students with 1.5B parameters or fewer, compared with 8.8 for students with 2B parameters or more.Larger students are more sensitive to quantization; Llama-3.2-3B adds 14.7 false alarms and loses 2.11 specificity points, versus 4.0 and 0.57 for Llama-3.2-1B.
  • MobileCLIP-based students: MobileCLIP-based students produce 4.7 and 2.7 false alarms but miss 39.3 and 14.7 fires, respectively.Their missed-fire counts make these configurations less suitable for practical fire detection despite low false-alarm rates.
Loading 2609.05782v1…