PaperlayerView exact revision on arXiv

Source-linked AI summary

IRSTD-Agent: Agentic Infrared Small Target Detection via Zoom-Guided Interaction Learning

Jiawen Xi, Yu Zhang, Tianyi Zhao, Zhu Liu, Maoxun Yuan, Xingxing Wei

arXiv:2610.05342v1cs.CVcs.AI

TL;DR

Infrared small-target detection matters for applications such as maritime monitoring and aerial surveillance, but existing MLLM-based approaches struggle to localize these targets precisely. IRSTD-Agent guides an MLLM through dynamic visual search with complementary tools and annotation-derived interaction supervision. It outperforms evaluated MLLM-based methods on WideIRSTD-Full and IRSTD-1K.

  • Problem

    Existing MLLM-based approaches struggle to precisely localize infrared small targets, which matter in applications such as maritime monitoring and aerial surveillance.

    Source: §1 · p. 1

  • Method

    IRSTD-Agent uses five complementary visual tools for dynamic search, while Zoom-guided Interaction Learning uses annotation-derived trajectories to supervise tool selection and arguments.

    Source: §1 · p. 2

  • Results

    IRSTD-Agent consistently outperforms existing MLLM-based methods and accurately localizes infrared small targets across scenarios on WideIRSTD-Full and IRSTD-1K.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work highlights actively acquiring fine-grained visual evidence as important for precise infrared small-target localization.

    Source: §5 · p. 9

    Sequential tool interactions add inference overhead, and extremely small or visually ambiguous targets remain challenging.

    Source: §5 · p. 9

Abstract

from arXiv · show

Infrared small-target detection plays an important role in maritime monitoring and aerial surveillance. Although multimodal large language models (MLLMs) offer promising capabilities for visual understanding, existing MLLM-based approaches struggle to precisely localize infrared small targets. In this paper, we propose IRSTD-Agent, an agentic framework for infrared small target detection through dynamic visual search. The framework enables an MLLM to adaptively determine where and at what scale to inspect an image and progressively gather fine-grained visual evidence for precise target localization. Five complementary visual tools (PROPOSAL, ZOOM, DETECT, DROP and REFINE) support object candidate discovery, adaptive observation, target localization, hypothesis rejection, and target extent refinement, together enabling a coordinated search process over original-resolution images. To teach the MLLMs to conduct this search, we introduce Zoom-guided Interaction Learning, which uses annotation-derived interaction trajectories to supervise tool selection and the corresponding arguments. Through extensive experiments on WideIRSTD-Full and IRSTD-1k datasets, we demonstrate that IRSTD-Agent outperforms the evaluated vision-language models and enhances the precise localization capabilities of MLLMs in IRSTD tasks.

1 INTRODUCTION

IRSTD-Agent frames infrared small-target detection as dynamic visual search, using visual tools and annotation-derived interaction trajectories to guide an MLLM. Experiments on WideIRSTD-Full and IRSTD-1K show it outperforms existing MLLM-based methods.

1 INTRODUCTION

  • Specialized IRSTD models may limit generalization, while existing MLLM adaptation approaches incur fine-tuning costs or have not established precise localization.Fine-tuning requires substantial annotation and computation; cue-guided prompting avoids parameter updates but its precise-localization effectiveness remains unestablished.Source: §1 · p. 1, Figure 1 · p. 2
  • The framework uses five visual tools for candidate discovery, adaptive observation, localization, clutter rejection, and target-extent refinement during dynamic search.Zoom-guided Interaction Learning uses annotation-derived interaction trajectories to supervise tool selection and arguments.Source: §1 · p. 2
  • IRSTD-Agent consistently outperforms existing MLLM-based methods and accurately localizes infrared small targets across various image scenarios.The reported evaluation covers WideIRSTD-Full and IRSTD-1K.Source: §1 · p. 2

2 RELATED WORK

Related work spans specialized infrared small-target detection, language-conditioned localization, and visual evidence acquisition through iterative reasoning or structured workflows.

2 RELATED WORK

  • Specialized IRSTD methods use tailored architectures and supervision, including feature propagation, scale- and location-sensitive supervision, and multi-scale feature interaction.Other work explores unified encoder–decoder architectures, target-adapted convolutions, sparse point annotations, and auxiliary acquisition metadata.Source: §2 · pp. 2–3
  • General-purpose MLLMs and grounding models support instruction-following localization or language-conditioned region detection and structured spatial prediction.Examples include Qwen3.5/3.8, InternVL3.5, MDETR, GLIP, LocateAnything, SenseNova-Vision, Grounding DINO, and Florence-2.Source: §2 · p. 3
  • Visual evidence acquisition methods use intermediate region interactions or structured workflows for grounding, fine-grained perception, and other visual tasks.Examples include iterative visual reasoning approaches, ZoomEye tree search, ZoomEarth adaptive cropping, and AgentDet’s shared-blackboard pipeline.Source: §2 · p. 3

3 IRSTD-AGENT

IRSTD-Agent treats infrared small-target detection as dynamic visual search over original-resolution images, using successive visual-tool interactions to gather evidence and localize targets. Annotation-derived interaction trajectories supervise tool choices and arguments.

3.1 AGENT FORMULATION

  • The agent conditions tool selection on the current view and interaction context, then executes each tool on the original image to update the search state.The state carries information such as candidate hypotheses, interaction history, accumulated detections, and remaining budget.Source: §3.1 · pp. 3–4

3.2 ACTIVE VISUAL TOOLS

  • PROPOSAL finds candidate locations, ZOOM acquires a new view, DETECT localizes visible targets, DROP rejects unsupported hypotheses, and REFINE estimates target extent.The tools operate on native-resolution pixels from the original image.Source: §3.2 · pp. 4–5
  • DROP ends a search branch when visual evidence does not support a target, records the rejection, and adds no image-level detection.Source: §3.2 · p. 5
  • REFINE suppresses local background, applies adaptive thresholding, and extracts the connected region around a detected center to obtain a refined box.The extracted region also defines a binary mask mapped back to original-image coordinates; if no valid region is found, the initialized box is retained and no mask is added.Source: §3.2 · pp. 5–6

3.3 ZOOM-GUIDED INTERACTION LEARNING

  • Supervised interaction actions pair each current view and accumulated context with the correct tool call, teaching both tool selection and its arguments.Prompt tokens are excluded from the supervised fine-tuning loss.Source: §3.3 · p. 6
  • Positive training trajectories zoom toward annotated targets before DETECT, while negative background hypotheses receive DROP supervision.Proposal-offset examples place targets away from the initial candidate location; views with multiple visible targets supervise DETECT to return all valid targets.Source: §3.3 · p. 6

4 EXPERIMENTS

Experiments evaluate independently on native-resolution WideIRSTD-Full and 512×512 IRSTD-1k, with ablations of tools, supervision, and action budget. IRSTD-Agent leads evaluated methods on both benchmarks, while qualitative examples show lingering difficulty with extremely small, visually ambiguous targets.

4.1 EXPERIMENTAL SETUP

  • WideIRSTD-Full tests 2,000 native-resolution images, while IRSTD-1k uses 800 training and 201 test images at 512×512 resolution.Evaluation matches box centers within 3 pixels and reports precision, recall, F1-scores, and mean IoU over matched targets.Source: §4.1 · pp. 6–7

4.2 ABLATION STUDIES

  • Without DROP, precision is 3.61% (recall 77.44%); adding it raises precision to 63.78% and F1-scores to 66.45%, while REFINE raises F1-scores to 77.97%.Fixed-scale observation instead yields 23.89% precision and 37.47% F1-scores, compared with 86.87% recall.Source: Table 1 · p. 7, §4.2 · p. 7
  • Full zoom-guided SFT achieves 81.09 precision and 77.97 F1-scores, outperforming the settings without SFT or without the negative branch.Under the same native tool-call protocol, w/o SFT scores 5.80 precision and 10.80 F1-scores; w/o negative branch scores 6.72 precision and 12.38 F1-scores.Source: Table 2 · p. 7, §4.2 · p. 7
  • B = 64 provides a favorable performance–budget trade-off; gains diminish as the model-call budget increases beyond that point.F1-scores rise by 0.87 points from 64 to 96 and 0.61 points from 96 to 128 despite substantially increased inference time.Source: Figure 4 · p. 7, §4.2 · p. 7

4.3 COMPARISON WITH SOTA METHODS

  • On WideIRSTD-Full, IRSTD-Agent achieves 46.87 F1-scores and 71.63 precision, a 37.72 F1-scores gain over direct Qwen3.5-27B inference using the same backbone, supporting active local observation in wide-area IRSTD.Source: §4.3 · p. 8, Table 3 · p. 8
  • On IRSTD-1k, IRSTD-Agent achieves 77.97 F1-scores, 81.09 precision, and 75.08 recall, exceeding the reported active-search alternatives.ZoomEye reaches 60.73 F1-scores and RSCoVLM-det reaches 75.13 F1-scores.Source: Table 3 · p. 8, §4.3 · pp. 8–9

4.4 QUALITATIVE ANALYSIS.

  • Representative examples show IRSTD-Agent recovering more weak targets and localizing more accurately, though extremely small, ambiguous targets remain difficult.The examples compare ground truth and predictions across both datasets, with target-containing regions enlarged in insets.Source: Figure 5 · p. 9, §4.4 · p. 9

5 CONCLUSION

IRSTD-Agent frames infrared small-target detection as dynamic visual search, using five tools and annotation-derived interaction trajectories to guide tool selection and arguments. Experiments on WideIRSTD-Full and IRSTD-1k show performance superior to other state-of-the-art MLLM-based methods, while sequential interactions add inference overhead and extremely small or ambiguous targets remain challenging.

5 CONCLUSION

  • The framework combines five visual tools for candidate discovery, adaptive observation, localization, hypothesis rejection, and extent refinement, with annotation-derived trajectories supervising tool choices and arguments.Experiments on WideIRSTD-Full and IRSTD-1k report detection performance superior to other state-of-the-art MLLM-based methods.Source: §5 · p. 9
  • Sequential tool interactions add inference overhead, while extremely small or visually ambiguous targets remain challenging.Future work could explore more efficient search policies and adaptive computation allocation to balance detection accuracy and computational cost.Source: §5 · p. 9

A.1 TOOLS AND COORDINATE INTERFACE

The appendix specifies how views and proposals are formed, how detections are refined and assembled, and how annotation-derived trajectories supervise search actions.

A.1 TOOLS AND COORDINATE INTERFACE

  • Each requested zoom size is the next view’s side length in original-image pixels, and views are extracted directly from the original image after rounding and boundary handling.The implementation represents views by boundary coordinates; tool execution updates the current view and branch-local history while preserving accumulated detections.Source: §A.1 · p. 13

A.2 PROPOSAL TOOL

  • PROPOSAL ranks and spatially suppresses peaks from fused spatial and frequency responses to form candidate locations, using image-based operations without learned parameters.The spatial branch combines normalized multi-scale top-hat, difference-of-Gaussians, local standardized-contrast, and patch-contrast responses; the frequency branch uses spectral-residual saliency.Source: §A.2 · p. 13

A.3 REFINEMENT TOOL AND OUTPUT ASSEMBLY

  • REFINE estimates local background, thresholds the residual adaptively, and extracts the center-associated connected region to define refined boxes and foreground masks.If no valid connected region is found, the initialized local box is retained; image-level masks preserve image-derived foreground regions rather than filling boxes.Source: §A.3 · pp. 13–14

A.4 ZOOM-GUIDED INTERACTION LEARNING

  • Annotation-derived trajectories supervise search actions: positive paths preserve targets for ZOOM and DETECT, while DROP labels apply only when no annotated target intersects the view.Prompt tokens are masked from the training loss, and separate trajectory sets and checkpoints are built for each dataset.Source: §A.4 · p. 14

A.5 BASELINE ADAPTATION

The baselines retain their original inference paradigms where possible, with full-image prompting for MLLMs and grounding models and adapted interfaces for visual-search methods.

A.5 BASELINE ADAPTATION

  • Released MLLM and grounding checkpoints are prompted for all visible infrared targets on the full image, with model-specific coordinates mapped to original-image coordinates.InternVL3.5, Qwen3.5/3.8, LocateAnything, and SenseNova-Vision are used without infrared-specific fine-tuning; grounding methods retain native interfaces where applicable.Source: §A.5 · p. 14
  • ZoomEye, ZoomEarth, DeepEyes, and VisionReasoner retain their search, cropping, or reasoning procedures while adapting task interfaces to produce infrared-target detections.ZoomEye uses Qwen3.5-27B and hierarchical visual search; predictions from local views are mapped to original-image coordinates.Source: §A.5 · p. 15
  • RSCoVLM-det is the only baseline in this group trained specifically for each dataset, using separate LoRA adapters and annotations converted from training masks.Source: §A.5 · p. 15
  • Specialized IRSTD detectors are trained separately on each dataset using their original supervision, with full-image and sliding-window evaluation protocols reported separately.Source: §A.5 · p. 15

B.1 COMPARISON WITH SPECIALIZED IRSTD METHODS

On large WideIRSTD-Full images, IRSTD-Agent exceeds specialized IRSTD detectors, although those detectors score higher on the full test set under their conventional input setting. Sliding windows recover much of their large-image performance, but remain below IRSTD-Agent.

B.1 COMPARISON WITH SPECIALIZED IRSTD METHODS

  • 54.55 recall and 64.29 F1: IRSTD-Agent exceeds every specialized detector on the 44-image Large subset, which contains 99 targets.IRSTD-Agent retains its active local-observation pipeline for this comparison.Source: Table 4 · p. 15, §B.1 · pp. 15–16
  • 54.64–60.81 F1 for specialized detectors exceeds IRSTD-Agent’s 46.87 F1 on the full test set.The full set contains 2,000 images and 5,328 targets.Source: §B.1 · p. 15
  • 1.01% target recovery and F1 around 1.9: all specialized detectors perform poorly on the Large subset under full-image resize.The comparison shows the dependence of specialized IRSTD models on observation scale; the sliding-window alternative preserves native resolution.Source: §B.1 · p. 15, Table 4 · p. 15
  • 45.88–50.57 F1 and 39.39–53.54 recall: native-resolution sliding windows substantially recover specialized-detector performance on the Large subset, but remain below IRSTD-Agent.The specialized detectors use native-resolution 256 × 256 windows with stride 128, without retraining.Source: §B.1 · pp. 15–16, Table 4 · p. 15

B.2 SEGMENTATION RESULTS

Segmentation evaluation uses a shared center-to-mask refinement procedure and reports overlap, target recovery, and false alarms. Results vary by dataset and metric, with higher target recovery sometimes accompanied by more false alarms.

B.2 SEGMENTATION RESULTS

  • All evaluated pipelines use the same center-to-mask refinement, so the full-image MLLM segmentation results include mask generation beyond their native-box outputs.Table 5 covers 2,000 WideIRSTD-Full images and 201 IRSTD-1k images; nIoU, Dice, and Pd are percentages, and Fa is reported as 106Fa.Source: §B.2 · p. 16, Table 5 · p. 16
  • The evaluation defines nIoU over images with nonempty mask unions, Dice over foreground pixels, Pd by centroid matching within three pixels, and Fa from unmatched predicted components.Lower Fa indicates fewer false alarms.Source: §B.2 · p. 16
  • On WideIRSTD-Full, IRSTD-Agent has the highest nIoU (31.71) and Dice (37.67), while Rex-Omni has higher Pd (38.95) but Fa 67.64 versus 13.35.On IRSTD-1k, IRSTD-Agent reaches 52.19 nIoU, 57.90 Dice, 76.09 Pd, and 17.61 Fa; ZoomEye has higher nIoU (54.36) and Pd (80.13), but Fa is 41.54.Source: §B.2 · pp. 16–17

C.1 IRSTD-AGENT

  • C.1 prompts specify crop-normalized action coordinates and use ZOOM for more local evidence, DETECT for visible targets, and DROP for unsupported hypotheses.PROPOSAL supplies candidate hypotheses rather than detections; DETECT and DROP end the current branch, after which the executor advances to another candidate.Source: §C.1 · pp. 17–18

C.2 FULL-IMAGE MLLM AND GROUNDING PROMPTS

  • C.2 full-image prompts request tight boxes for compact targets, while grounding methods retain their native task formats and output parsers.Qwen coordinates are independently normalized to [0, 1000] and mapped back to original-image coordinates.Source: §C.2 · p. 18

D LARGE LANGUAGE MODEL USAGE

  • D discloses AI assistance for language polishing, LaTeX formatting, and portions of code implementation and modification.The authors state that they take full responsibility for the final manuscript, code, and reported results.Source: D LARGE LANGUAGE MODEL USAGE · p. 18
Loading 2610.05342v1…