PaperlayerView exact revision on arXiv

Source-linked AI summary

IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting

Tao Zhang, Yuyang Hong, Yang Xia, Kun Ding, Zeyu Zhang, Ying Wang, Shiming Xiang, Chunhong Pan

arXiv:2512.09663v1cs.CV

TL;DR

MLLMs’ infrared-image understanding remains underexplored, despite infrared imagery’s use in low-light and adverse-weather settings. This paper introduces IF-Bench and training-free GenViP, which combines aligned RGB translations with original infrared inputs. GenViP improves performance across diverse models, while the benchmark identifies persistent difficulty with fine-grained infrared understanding.

  • Problem

    Infrared-image understanding by MLLMs remains unclear because prior studies have narrow task coverage, lack human calibration, and evaluate few models.

    Source: §1 · p. 1

  • Method

    The paper constructs IF-Bench and evaluates over 40 MLLMs, then proposes training-free GenViP, which jointly feeds aligned RGB translations and original infrared images to models.

    Source: §1 · pp. 1–2, §4.1 · p. 5

  • Results

    GenViP consistently improves diverse MLLMs on IF-Bench, achieving up to 7% relative performance gain.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The evaluation finds that current MLLMs still perform poorly on fine-grained infrared understanding, while GenViP provides evidence that generated visual prompts can facilitate visual understanding.

    Source: §6 · p. 8

    IF-Bench contains relatively few images and questions and omits some more challenging task types.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in multimodal large language models (MLLMs) have led to impressive progress across various benchmarks. However, their capability in understanding infrared images remains unexplored. To address this gap, we introduce IF-Bench, the first high-quality benchmark designed for evaluating multimodal understanding of infrared images. IF-Bench consists of 499 images sourced from 23 infrared datasets and 680 carefully curated visual question-answer pairs, covering 10 essential dimensions of image understanding. Based on this benchmark, we systematically evaluate over 40 open-source and closed-source MLLMs, employing cyclic evaluation, bilingual assessment, and hybrid judgment strategies to enhance the reliability of the results. Our analysis reveals how model scale, architecture, and inference paradigms affect infrared image comprehension, providing valuable insights for this area. Furthermore, we propose a training-free generative visual prompting (GenViP) method, which leverages advanced image editing models to translate infrared images into semantically and spatially aligned RGB counterparts, thereby mitigating domain distribution shifts. Extensive experiments demonstrate that our method consistently yields significant performance improvements across a wide range of MLLMs. The benchmark and code are available at https://github.com/casiatao/IF-Bench.

1. Introduction

IF-Bench addresses the largely unexplored question of how well MLLMs understand infrared images with a curated benchmark and a broad evaluation. The paper also introduces training-free GenViP, which uses aligned RGB counterparts alongside infrared inputs to improve performance.

Motivation

  • Infrared understanding remains unclear because prior evaluations have narrow task coverage, lack human calibration, and assess few models.Infrared images are an out-of-domain input for MLLMs primarily trained on natural images.Source: §1 · p. 1

IF-Bench

  • IF-Bench contains 499 infrared images and 680 curated VQA pairs across 10 dimensions, built from images sourced from 23 datasets.Questions span coarse-grained perception, fine-grained perception, and image reasoning.Source: §1 · pp. 1–2

Evaluation

  • Evaluation of over 40 MLLMs finds scale-related gains, an accuracy-efficiency trade-off for MoE, mixed effects of thinking mode, and comparable open- and closed-source performance.Models still struggle with fine details in infrared images.Source: §1 · p. 2

GenViP

  • GenViP consistently improves diverse MLLMs on IF-Bench, with up to 7% relative gain and performance exceeding some closed-source understanding models.The method uses translated RGB counterparts jointly with original infrared inputs; its editing model is fine-tuned on 50,000 RGB-T pairs to surpass named closed-source editing models.Source: §1 · p. 2

2. Related Work

Prior multimodal benchmarks cover broad visual tasks, specialized knowledge, temporal perception, or spatial reasoning, while infrared evaluation remains limited. IF-Bench responds with diverse curated tasks and evaluation across more than 40 mainstream MLLMs.

MLLMs

  • Prior MLLM work advances visual reasoning through approaches including dynamic resolution, cascade reinforcement learning, thinking mode, and Mixture-of-Experts design.The cited examples include Qwen2.5-VL, InternVL3.5, and Qwen3-VL.Source: §2 · p. 2

Infrared Adaptation

  • Infrared-specific fine-tuning efforts face scarce image-text data, complex adaptation pipelines, and difficulty keeping pace with evolving MLLMs.GenViP instead requires no training or image-text data and is described as directly applicable to any MLLM.Source: §2 · p. 3

Multimodal Understanding Benchmark

  • Earlier infrared benchmarks built automatically from annotated datasets have limited task coverage, little manual verification, and few evaluated models.IF-Bench is presented as a diverse, carefully curated benchmark evaluated on more than 40 mainstream MLLMs.Source: §2 · p. 3

3. IF-Bench

IF-Bench defines infrared understanding tasks, constructs a curated VQA benchmark, and uses multiple evaluation strategies intended to improve reliability.

3.1. Task Definition

  • Infrared understanding is divided into coarse-grained perception, fine-grained perception, and image reasoning, further organized into ten dimensions.The dimensions include scene and theme recognition, localization, counting, thermal-feature understanding and reasoning, and commonsense reasoning.Source: §3.1 · pp. 3–4

3.2. Construction Pipeline

  • Image collection filters an initial 2,300-image pool to 1,166 high-quality images, then combines manual annotation and automatic question generation to produce 4,628 initial VQA pairs.Qwen2.5-VL-72B generates at most four single-choice questions per image; manual annotation supplies descriptions and bounding boxes for target localization.Source: §3.2 · p. 4
  • Two-stage human calibration checks plausibility, ambiguity, answerability, correctness, and difficulty, and adds VQA pairs; the final benchmark has 499 images and 680 bilingual pairs.The fine-grained filtering stage is performed by domain experts in infrared imaging.Source: §3.2 · pp. 4–5

3.3. Evaluation Protocol

  • Evaluation averages scores across all four cyclic option permutations and both languages, with exact matching followed by Qwen3-7B parsing when needed.The protocol also uses a unified prompt; circular and bilingual evaluation makes each question count eight times toward evaluation.Source: §3.3 · p. 5

4. Infrared Understanding Enhancement

GenViP pairs translated RGB images with the original infrared input to address distribution shifts without model fine-tuning. The authors also fine-tune an open-source editing model on RGB-T pairs, improving its infrared-to-RGB translation quality.

4.1. Generative Visual Prompting

  • GenViP translates infrared images into spatially and semantically aligned RGB images, then feeds both images to MLLMs to reduce distribution shifts while retaining thermal information.This inference-time approach avoids requiring infrared image–text datasets or fine-tuning each MLLM.Source: §4.1 · p. 5
  • Adding a brief textual prior about infrared characteristics to the prompt further enhances interpretation of thermal features.Experiments also validate GenViP's effectiveness across MLLMs.Source: §4.1 · p. 5

4.2. Editing Model Optimization

  • From over 370,000 collected RGB-T pairs, filtering and balanced sampling produce a 50,000-pair dataset for editing-model fine-tuning.The filtering screens for alignment, resolution, pairing quality, and potential overlap with IF-Bench.Source: §4.2 · pp. 5–6
  • Fine-tuned Qwen-Edit-2509-FT achieves superior infrared-to-RGB translation quality and outperforms closed-source editing models on IF-Bench.The authors report that translation quality strongly affects GenViP's performance and that the untuned open-source Qwen-Edit-2509 trails leading closed-source editors.Source: §4.2 · pp. 5–7

5. Experiments

Experiments evaluate diverse MLLMs on IF-Bench and find that GenViP improves performance, with the strongest reported score reaching 84.4. Results also show that model scale, reasoning mode, input configuration, and evaluation dimension relate to performance.

5.1. Evaluation Models.

  • The evaluation covers over 40 mainstream MLLMs on IF-Bench, spanning 1B to 241B parameters, dense and MoE architectures, and models with and without explicit reasoning capabilities.Source: §5.1 · p. 7

5.2. Evaluation Results of IF-Bench.

  • Architecture gaps are largest among smaller models and narrow at larger scales; model size is positively correlated with performance across dimensions.Qwen3-VL-8B scores 78.8, exceeding Qwen2.5-VL-72B and approaching InternVL3-78B with about one-tenth the parameters.Source: §5.2 · p. 7
  • For object counting and spatial relationship understanding in infrared image understanding, average accuracies fall below 60, and the highest scores are only around 70, indicating challenges in capturing fine-grained details.Source: §5.2 · p. 8
  • 0.34 points: thinking yields only a modest average gain, helping thermal feature tasks while reducing accuracy in target localization and action recognition.Its effects vary across model families, and the authors find the added reasoning overhead makes it ineffective as an overall enhancement approach.Source: §5.2 · p. 7, Figure 5 · p. 7
  • 0.3 points separate the best closed-source model from the best open-source model; three other closed-source models score below the leading open-source models.The reported results indicate a minimal open- versus closed-source performance gap.Source: §5.2 · p. 7

5.3. Results of GenViP

  • 84.4: Qwen3-VL-235B-A22B-Instruct with GenViP outperforms every model in Tab. 2, while the 8B variant matches 78B and 106B models.GenViP improves performance across models, with gains diminishing as model scale increases.Source: §5.3 · p. 8, Table 4 · p. 8
  • Fine-tuned Qwen-Edit-2509-FT bridges the gap with closed-source editing models and surpasses them on IF-Bench, while the untuned open-source editor lags behind.All tested editing models enable GenViP to improve infrared understanding.Source: §5.3 · p. 8, Table 5 · p. 8
  • Combining infrared and translated RGB inputs improves performance across most dimensions, while adding a textual prior further boosts thermal understanding and reasoning.RGB-only input loses thermal information and reduces performance on several thermal tasks; the combined inputs plus textual prior yield the best overall performance.Source: §5.3 · p. 8, Table 6 · p. 8

6. Conclusion

IF-Bench provides a benchmark for infrared image understanding and an evaluation of over 40 MLLMs. The paper reports several performance patterns, proposes GenViP, and notes limits in benchmark size and task coverage.

6. Conclusion

  • Evaluation of over 40 MLLMs finds that scale and architecture influence performance, thinking does not effectively improve overall results, open- and closed-source models perform comparably, and fine-grained infrared understanding remains difficult.Source: §6 · p. 8
  • GenViP uses generative models to produce visual prompts and significantly improves existing MLLMs' infrared understanding capability.The authors present this as evidence that generation can facilitate visual understanding.Source: §6 · p. 8
  • The benchmark has relatively few images and questions and omits some more challenging task types; future work will expand and diversify it.Source: §6 · p. 9

A. Details of IF-Bench

The appendix describes IF-Bench's examples and image distribution across its constituent sub-datasets. It reports relatively even distribution after manual filtering, with lower counts in some datasets attributed to homogeneous scenes.

A. Details of IF-Bench

B. Details of Prompts

The appendix provides the prompts used for evaluation, LLM-based parsing, GenViP dual-image evaluation and textual priors, and automatic VQA generation. Only English versions are shown; the Chinese versions are described as semantically equivalent.

B. Details of Prompts

  • The appendix presents prompts for MLLM evaluation, LLM-based parsing, GenViP dual-image evaluation, textual priors, and automatic VQA generation.Only English prompt versions are shown; the Chinese versions are described as semantically equivalent.Source: B. Details of Prompts · p. 12
  • The unified system prompt is used when evaluating MLLMs on IF-Bench.Source: B. Details of Prompts · p. 12
  • The listed prompts also cover LLM-based parsing and automatic VQA generation.Source: B. Details of Prompts · p. 12

C. Details of Editing Models

This section specifies the editing models’ inference setup, Qwen-Edit-2509 fine-tuning, and the shared editing prompt used for training and inference.

C. Details of Editing Models

  • Seedream 4.0 and Gemini-2.5-Flash use official APIs, while Qwen-Edit-2509 and its fine-tuned version run with 40 inference steps.Qwen-Edit-2509 is deployed locally.Source: C. Details of Editing Models · p. 12
  • Qwen-Edit-2509-FT is fine-tuned with LoRA at rank 32 for two epochs on 50,000 RGB-T pairs.Training uses batch size 8 and fixed resolution 1024.Source: C. Details of Editing Models · p. 12
  • Training and inference use the same prompt: translate the infrared image into its corresponding visible-light RGB image.More sophisticated prompts yielded similar results, so the authors use the simple prompt throughout.Source: C. Details of Editing Models · p. 12

D. More Analyses

The supplementary analyses report language-score patterns, sensitivity to option order, and differences in translated-image quality across editing models. They also relate GenViP to Thinking-with-Image and identify autonomous editing-model selection as future work.

Language Preference

Stricter Correctness Judgement

  • Under the all-option-orders correctness criterion, average scores drop by more than 10 points for several models, indicating limited robustness to option-order variations.The stricter criterion counts a question as correct only when the model answers correctly under every option order.Source: D. More Analyses · p. 12, Table 7 · p. 14

Relationship with Thinking-with-Image

  • GenViP shares Thinking-with-Image’s intuition of processing images to improve understanding, but explicitly requires an editing-model call at the initial stage.The authors suggest extending the paradigm to let models decide autonomously whether to call an editing model, leaving this for future work.Source: D. More Analyses · p. 12

More Analyses

Visualization of Translated Quality

  • Fine-tuning Qwen-Edit-2509 on 50,000 infrared–RGB pairs improves translation naturalness and spatial and semantic consistency, yielding a substantial GenViP performance gain.Closed-source models show strong generalization on some inputs but can produce unnatural colors or imperfect spatial correspondence; Qwen-Edit-2509 performs worse overall than those models.Source: D. More Analyses · pp. 12–13, Figure 15 · p. 17
Loading 2512.09663v1…