PaperlayerView exact revision on arXiv

Source-linked AI summary

Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation

Chao Hao, Jun Xu, Ji Du, Shuo Ye, Ziyue Qiao, Xiaodong Cun, Guangcong Wang, Xubin Zheng, Zitong Yu

arXiv:2605.12953v1cs.CVcs.AI

TL;DR

Language-guided segmentation aims to produce masks for text-specified targets, but MLLMs can struggle to ground prompts spatially and text-only reasoning lacks visual feedback. Seg-Agent uses iterative visual reasoning at test time without training, and the authors report performance comparable to training-based methods. Various-LangSeg evaluates generalization across three segmentation scenarios.

  • Problem

    MLLMs can generate low-quality visual prompts because of limited spatial grounding, while text-only reasoning does not directly verify predictions against visual evidence.

    Source: §1 · pp. 1–2

  • Method

    Seg-Agent uses a training-free test-time chain of generation, selection, and refinement, with visual prompts rendered on the image for multimodal reasoning.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    Seg-Agent achieves segmentation performance comparable to training-based methods while remaining completely training-free.

    Source: §1 · p. 3, §6 · p. 11

  • Takeaways & Limitations

    Various-LangSeg provides a unified benchmark for evaluating generalization across explicit semantic, generic object, and reasoning-guided segmentation.

    Source: §1 · p. 2, §6 · p. 11

    Seg-Agent requires multiple MLLM calls during inference, adding latency compared with single-step approaches.

    Source: §5.4 · p. 11

Abstract

from arXiv · show

Language-guided segmentation transcends the scope limitations of traditional semantic segmentation, enabling models to segment arbitrary target regions based on natural language instructions. Existing approaches typically adopt a two-stage framework: employing Multimodal Large Language Models (MLLMs) to interpret instructions and generate visual prompts, followed by foundational segmentation models (e.g., SAM) to produce masks. However, due to the limited spatial grounding capabilities of off-the-shelf MLLMs, these methods often rely on extensive training on large-scale datasets to achieve satisfactory accuracy. While recent advances have introduced reasoning mechanisms to improve performance, they predominantly operate within the textual domain, performing chain-of-thought reasoning solely based on abstract text representations without direct visual feedback. In this paper, we propose Seg-Agent, a completely training-free framework that pioneers Explicit Multimodal Chain-of-Reasoning. Unlike prior text-only reasoning, our approach constructs an interactive visual reasoning loop comprising three stages: generation, selection, and refinement. Specifically, we leverage Set-of-Mark (SoM) visual prompting to render candidate regions directly onto the image, allowing the MLLM to ``see'' and iteratively reason about spatial relationships in the visual domain rather than just the textual one. This explicit multimodal interaction enables Seg-Agent to achieve performance comparable to state-of-the-art training-based methods without any parameter updates. Furthermore, to comprehensively evaluate generalization across diverse scenarios, we introduce Various-LangSeg, a novel benchmark covering explicit semantic, generic object, and reasoning-guided segmentation tasks. Extensive experiments demonstrate the effectiveness and robustness of our method.

1 Introduction

Language-guided segmentation supports arbitrary text-specified targets, but MLLMs often produce low-quality prompts because of limited spatial grounding, and text-only reasoning lacks direct visual verification. Seg-Agent uses test-time visual reasoning, while Various-LangSeg broadens evaluation across three segmentation scenarios.

1 Introduction

  • Existing two-stage methods use MLLMs to generate prompts for a segmentation model, but weak spatial grounding can make those prompts low quality.Training-based approaches address prompt quality, while text-only reasoning still lacks direct interaction with visual evidence.Source: §1 · pp. 1–2
  • Seg-Agent iteratively generates, selects, and refines visual prompts by rendering candidate boxes on the image for multimodal reasoning at test time.Visual feedback lets the model compare candidates, identify misalignments, and refine prompts without training.Source: §1 · p. 2, Figure 1 · p. 2
  • Various-LangSeg evaluates explicit semantic, generic object, and reasoning-guided segmentation, spanning named categories, general concepts, and targets requiring abstract reasoning.Examples include “cat,” “camouflaged object,” and “carbohydrate-rich food.”Source: §1 · p. 2, Figure 2 · p. 3
  • Seg-Agent is reported to achieve performance comparable to training-based methods while remaining completely training-free.The authors also describe the approach as a low-cost design paradigm.Source: §1 · p. 3

2 Related Work

Prior work combines MLLMs and foundational segmentation models, but fine-grained spatial grounding remains challenging. Seg-Agent focuses on improving MLLM-generated visual prompts through an explicit reasoning chain without training.

2.1 Multimodal Large Language Models

  • MLLMs perform multimodal tasks but typically lack the fine-grained spatial perception and grounding needed for segmentation.Prior approaches often improve grounding by training on task-specific data; Seg-Agent instead uses an explicit multimodal reasoning chain.Source: §2.1 · p. 3

2.2 Foundational Segmentation Models

  • SAM offers a promptable interface that accepts sparse cues such as points or boxes and provides general-purpose mask prediction.The framework uses SAM2 and targets the quality of MLLM-generated prompts.Source: §2.2 · pp. 3–4

2.3 Language-Guided Segmentation

  • Language-guided segmentation methods variously generate masks with decoders, use a special token, or post-train MLLMs to produce prompts SAM can interpret.Seg-Agent is closest to SAM4MLLM and Seg-Zero, but improves prompts with an explicit reasoning chain without training.Source: §2.3 · p. 4

3 Various-LangSeg: A Comprehensive Evaluation Benchmark

Existing benchmarks cover a limited range of language-guided segmentation tasks, motivating Various-LangSeg as a unified evaluation benchmark. It prioritizes broad scenario coverage and assesses generalization across three task types.

3.1 Motivation

  • Earlier language-guided segmentation datasets focus mainly on referring and reasoning segmentation, leaving limited coverage for evaluating general-purpose models.Various-LangSeg is introduced as a unified benchmark spanning broader scenarios.Source: §3.1 · p. 4

3.2 Task Categorization

  • The benchmark groups tasks into explicit semantic segmentation, generic object segmentation, and reasoning-guided segmentation.These cover direct category names, general concepts, and queries requiring abstract reasoning, respectively.Source: §3.2 · p. 4

3.3 Dataset Construction

  • Each dataset item pairs an image and mask with a manually annotated textual instruction, after selection and review of image-mask pairs from public datasets.The authors describe this process as more than simple sampling and assembly.Source: §3.3 · p. 4
  • Various-LangSeg contains 244 evaluation samples, prioritizing scenario diversity over scale to stress-test generalization.The samples cover the three scenarios, with 140, 64, and 40 samples, respectively.Source: §3.3 · pp. 4–5

3.4 Evaluation Metrics

  • Evaluation uses gIoU, the average IoU over samples, and cIoU, the ratio of total intersection area to total union area across the dataset.Source: §3.4 · p. 5

4 Method

Seg-Agent is a training-free, two-stage language-guided segmentation framework that uses test-time multimodal reasoning to guide visual-prompt generation. Its generation, selection, and refinement steps make prompt decisions explicit before a base segmentation model produces the mask.

4.1 Problem Formulation

  • Seg-Agent combines an MLLM-generated visual prompt with a base segmentation model to turn a natural-language instruction into a binary target mask.The task is to segment the object described by the instruction.Source: §4 · p. 5, §4.1 · p. 5

4.2 Overview of Seg-Agent Pipeline

  • The pipeline proposes diverse boxes, selects the most appropriate one through visual comparison, refines its boundaries, and passes it to a pretrained segmenter such as SAM.This keeps the prompt-generation process explicit and stepwise rather than relying on learned end-to-end features.Source: §4.2 · p. 5, §4 · p. 5

4.3 The Forward Pass of Seg-Agent

  • Image augmentations such as flipping and scaling yield multiple MLLM-generated box proposals that are consolidated in the original image frame.Each augmented image is paired with the text instruction before proposal generation.Source: §4.3 · p. 5
  • Set-of-Mark prompting renders candidate boxes on the original image so the MLLM can compare their spatial relationship to the target and select a box.The selected box is then refined using semantic and visual context before serving as the segmentation model’s prompt.Source: §4.3 · pp. 5–6
  • The refined box guides SAM to produce a pixel-level binary mask, completing the transformation from language instruction to segmentation.The paper attributes improved final accuracy to the progressively refined visual prompts.Source: §4.3 · p. 6

4.4 Why Seg-Agent Matters

  • Seg-Agent operates without task-specific training or parameter updates, with explicit traceable steps and the ability to adapt to newer MLLMs or segmentation models without retraining.The authors also describe augmentation-enriched proposals as supporting robustness across unseen distributions.Source: §4.4 · p. 6

5 Experiments

Experiments compare Seg-Agent with training-based and training-free methods across referring, reasoning-guided, and three-scenario benchmark tasks. The results report improvements over the Qwen2.5VL+SAM2-L baseline, module gains, and trade-offs in inference cost and benchmark scale.

5.2 Comparison With Other Methods

  • On referring segmentation datasets, Seg-Agent improves on Qwen2.5VL+SAM2-L and performs comparably to Seg-Zero, while methods generally perform well with little difference between traditional and MLLM-based approaches.Table 1 reports cIoU for refCOCO, refCOCO+, and refCOCOg.Source: §5.2 · p. 7, Table 1 · p. 7
  • On Various-LangSeg, Seg-Agent improves on the Qwen2.5-VL+SAM2-L baseline in all three scenarios, and Seg-Agent-7B is the strongest training-free method, outperforming many training-based approaches.The benchmark reports performance across explicit semantic, generic object, and reasoning-guided scenarios, plus overall performance.Source: Table 3 · p. 8, §5.2 · p. 10
  • On ReasonSeg, Seg-Agent improves over the Qwen2.5VL+SAM2-L baseline and performs comparably to training-based Seg-Zero, surpassing earlier approaches such as LISA.The task uses reasoning-based expressions, including long and short queries, and is described as more challenging than explicit semantic segmentation.Source: §5.2 · pp. 7–8
  • Across language and image forms, the supplementary visualizations include Chinese and English inputs and real-world, captured, cartoon, and AI-generated images.The Fig. 5 caption presents these as examples of Seg-Agent’s generalization and application scenarios.Source: Figure 5 · p. 9
  • Visual comparisons show Seg-Agent accurately segments a sunflower, preserves generic-object structure without blurred edges, and identifies the intended person in a reasoning-guided example.The comparisons include LISA, Seg-Zero, and Qwen2.5-VL-7B+SAM2-L.Source: Figure 4 · p. 8, §5.2 · p. 10

5.3 Ablation Study

  • The full generation-selection-refinement pipeline performs best across scenarios, with a +5.1 gIoU gain over baseline on reasoning-guided segmentation; GM+RM can partly compensate for proposal errors without explicit selection.The ablation uses a fixed Qwen2.5-VL-7B + SAM2-L backbone; GM, SM, and RM denote generation, selection, and refinement modules.Source: §5.3 · p. 10, Table 4 · p. 10
  • Seg-Agent adapts across different MLLMs and segmentation models, and performance improves with stronger base models.Table 5 compares base-model combinations and Seg-Agent configurations against direct single-step inference.Source: Table 5 · p. 10, §5.3 · p. 11

5.4 Analysis and Limitations

  • The method shifts resource use from training to inference: it adds latency through multiple MLLM calls, while its total output-token count is comparable to text chain-of-reasoning methods.The authors also note that the 244-sample Various-LangSeg benchmark prioritizes coverage across scenarios over scale.Source: §5.4 · p. 11

6 Conclusion

Seg-Agent uses a training-free test-time multimodal reasoning chain to guide more accurate visual prompts and achieves performance comparable to training-based methods. Various-LangSeg provides a multi-scenario benchmark for evaluating generalization in language-guided segmentation.

6 Conclusion

  • Seg-Agent achieves segmentation performance comparable to training-based methods using a training-free chain of generation, selection, and refinement.The chain guides the model in generating more accurate visual prompts.Source: §6 · p. 11
  • Various-LangSeg contains multiple scenarios and is designed to comprehensively evaluate language-guided segmentation models’ generalization capability.Source: §6 · p. 11
  • The authors report that extensive experiments demonstrate the effectiveness of Seg-Agent.Source: §6 · p. 11

Related papers

Loading 2605.12953v1…