InstructSAM: Segment Any Instance with Any Instructions

Yuqian Yuan; Wentong Li; Zhaocheng Li; Yutong Lin; Juncheng Li; Siliang Tang; Jun Xiao; Yueting Zhuang; Wenqiao Zhang

Source-linked AI summary

InstructSAM: Segment Any Instance with Any Instructions

Yuqian Yuan, Wentong Li, Zhaocheng Li, Yutong Lin, Juncheng Li, Siliang Tang, Jun Xiao, Yueting Zhuang, Wenqiao Zhang

arXiv:2605.26102v2May 25, 2026cs.CV

TL;DR

Complex instructions require models to jointly understand language, ground targets, and separate instances, while existing approaches can be slow, brittle, or duplicate predictions. InstructSAM addresses this with parallel instance-aware query slots linking a vision-language model to SAM3, and its compact 2B-scale model achieves strong performance across instruction-driven and referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3’s agentic pipeline.

Problem
Instruction-driven segmentation must jointly handle language understanding, visual grounding, and instance separation for implicit intents under open-vocabulary settings.
Method
InstructSAM translates arbitrary instructions into parallel learnable instance slots and projects them into SAM3’s detector query space for multi-instance segmentation.
Results
2B-scale InstructSAM consistently achieves strong performance across instruction-driven and referring segmentation benchmarks, substantially outperforming prior end-to-end methods and SAM3’s agentic pipeline.
Takeaways & Limitations
InstructSAM provides accurate and efficient multi-instance masks for complex free-form instructions in a single unified framework.
Takeaways & Limitations
Existing complex-instruction approaches may be slow and brittle because prompt rewriting can lose fine-grained constraints and iterative filtering can accumulate errors.

Abstract

from arXiv · show

In this paper, we introduce InstructSAM, a unified and streamlined framework designed for multi-instance segmentation under arbitrary instructions. We formulates instruction-driven instance segmentation as a set-structured query prediction problem and propose an explicit reasoning-to-instance query interface that elegantly bridges a vision-language model (VLM) and SAM3. Specifically, a bank of learnable instance queries is injected into the VLM and contextualized with instruction and visual information, enabling each query to serve as an instance-aware slot. A hybrid-attention mechanism further promotes interaction among these queries, visual tokens, and instruction tokens, improving instance enumeration and reducing duplicate predictions. The resulting LLM-conditioned queries are projected into SAM3's detector query space to drive accurate multi-instance segmentation in a single forward pass. This design equips SAM3 with high-level instruction understanding, compositional reasoning, and instance-level set prediction without modifying its core architecture. To support training and evaluation, we further construct Inst2Seg, a high-quality and large-scale instruction-based instance segmentation dataset and benchmark that couples free-form instructions with instance-level masks. Extensive experiments show that only 2B-scale InstructSAM achieves strong results across complex instruction-driven and phrase-level referring segmentation benchmarks, outperforming prior end-to-end methods and SAM3's agentic pipeline while enabling efficient single-pass multi-instance prediction.

1 Introduction

The introduction frames complex instruction-driven multi-instance segmentation as a limitation of existing pipelines and token-based interfaces. InstructSAM addresses this with an explicit reasoning-to-instance query interface and introduces Inst2Seg to support training and evaluation.

Limitations: Existing agentic decomposition-and-filtering pipelines repeatedly invoke SAM3 and post-filter candidate masks, making complex-instruction segmentation slow, brittle, and error-prone.These systems rewrite instructions into concept-level prompts before iterative mask generation and heuristic or prompt-based verification.
Limitations: Token-as-mask interfaces support reasoning-driven semantic segmentation but are not inherently instance-discriminative for multi-instance prediction.LISA++ emits multiple [SEG] tokens, yet the interface remains limited for consistently separating instances.
InstructSAM: InstructSAM translates complex instructions into set-structured, instance-aware query representations that explicitly interface with SAM3 for coherent multi-instance segmentation.The framework uses LLM reasoning to interpret instructions rather than directly generating masks token by token.
Inst2Seg: Inst2Seg contains 500K QA pairs and a benchmark with 3,328 manually verified instructions spanning single-target, multi-target, and no-target scenarios.The dataset couples free-form instructions with instance-level masks and covers diverse real-world scenarios and instruction types.
Results: 2B-scale InstructSAM substantially outperforms prior end-to-end methods and SAM3’s agentic pipeline across established and newly introduced benchmarks.The introduction presents this as an outcome of extensive experiments.

2 Related Work

Prior work progresses from promptable generic segmentation toward instruction-driven pixel grounding, but existing approaches remain limited in handling complex compositional instructions or predicting multiple instances directly. MLLM-based methods commonly use segmentation-token interfaces, with LISA++ extending this paradigm through multiple tokens and bipartite matching.

Promptable Segmentation: SAM introduced prompt-to-mask segmentation, while SAM3 extended promptable segmentation to open-world, concept-level multi-instance retrieval from short noun phrases.SAM accepts points, boxes, or coarse masks; SAM3 enables multiple-object segmentation from concise concepts.
Promptable Segmentation: SAM3 is not designed for complex compositional instructions involving reasoning, exclusion, or counting, and SAM3-I addresses this with an instruction-aware adapter requiring model modification and retraining.SAM3-I maps natural-language instructions to masks by adapting SAM3.
MLLM-Based Pixel Grounding: MLLM segmentation methods typically project a special segmentation-token embedding, such as <SEG>, into a mask decoder’s prompt space to produce pixel-level masks end to end.This embedding-as-mask interface aligns phrase-level semantics with pixel outputs.
MLLM-Based Pixel Grounding: Most token-based methods emit one segmentation token per grounded region, while LISA++ predicts multiple <SEG> tokens and uses bipartite matching for instance assignment during training.LISA++ extends single-region semantic grounding toward multi-instance prediction.
MLLM-Based Pixel Grounding: X-SAM pursues a broader “any segmentation” formulation by standardizing textual prompts with phrase delimiters.It is presented alongside LISA++ as a parallel effort toward broader or multi-instance segmentation.

3 Method

InstructSAM formulates instruction-driven instance segmentation as set-structured query prediction, combining multimodal instruction reasoning with parallel instance slots and SAM3 mask decoding. Its hybrid attention, phrase conditioning, and multi-task training support open-form, multi-instance prediction in a single forward pass.

Task formulation: Instruction-driven instance segmentation predicts a variable-size set of instance masks from an image and open-form instruction, requiring language understanding, visual grounding, separation, and enumeration.Instructions may specify categories, referring phrases, attributes, relations, counting, exclusion, or implicit intent; the number of selected instances can be zero.
Framework overview: InstructSAM combines a multimodal LLM, parallel learnable mask queries, and a SAM3-based set-prediction decoder for multi-instance localization and mask decoding.The query bank explicitly parameterizes instance slots as the interface between instruction reasoning and downstream mask prediction.
Instance-query interface: Learnable queries replace autoregressive segmentation tokens and are inserted into the multimodal sequence, where each contextualized query becomes a grounded instance hypothesis.Each query integrates instruction semantics, global visual context, and query-level interactions to encode both what and where to segment.
SAM3 decoding: Projected query embeddings and phrase features condition SAM3’s visual features and detector, producing multiple instance masks in a single forward pass.The LLM also generates a short target phrase that stabilizes decoder conditioning and reduces ambiguity for implicit-intent or multi-step instructions.
Hybrid attention: Hybrid attention gives text tokens causal attention while allowing instance queries to attend bidirectionally to visual, textual, and other query tokens.This global-context design promotes stable, instance-discriminative mask prediction and supports set-level coherence.
Training objectives: End-to-end training jointly optimizes a multi-task objective including a masked autoregressive loss that excludes special segmentation-related tokens from language modeling.The masked objective uses a binary mask to prevent instance query tokens and <mask_end> from contributing to the language-modeling loss.

4 Inst2Seg Dataset

Inst2Seg is a large-scale instruction-based instance segmentation dataset and benchmark coupling free-form instructions with instance-level masks for fine-grained reasoning and precise annotation. Its benchmark evaluates single-target, multi-target, and no-target cases across object- and part-level instructions using instance- and semantic-level metrics.

Dataset design: Inst2Seg couples free-form instructions with instance-level masks to support complex instruction-driven segmentation.The dataset is designed for fine-grained instruction reasoning and precise mask annotation.
Training Data: Training images combine exo-centric data from SA-1B and COCO2017 with ego-centric data curated from Ego4D, EPIC-KITCHENS, and HD-EPIC.Ego-centric clips are selected for substantial scene variation, while blurry or low-quality frames are discarded.
Training Data: The annotation pipeline generates localization-oriented questions, hard negatives, noun-phrase answers, and explicit counting or quantifier fields for multi-instance targets.It also consolidates objects and generates boxes as part of the annotation process.
Benchmark: 986 images and 3,328 unique instructions define a benchmark spanning single-target, multi-target, and no-target cases at object- and part-level granularity.All benchmark instructions and masks are manually verified.
Metrics: mAP is the primary metric, with results stratified by target count and generalized IoU reported as a complementary semantic-level metric.The stratification covers single-target, multi-target, and no-target cases.

5 Experiments

InstructSAM-2B performs strongly across instruction-driven, phrase-level, and robotic segmentation benchmarks, while ablations show that learnable queries, hybrid attention, alignment, and filtered supervision are central to its performance. Increasing the query bank offers minimal gains but raises inference cost, supporting efficient single-pass prediction with a small query set.

Inst2Seg: InstructSAM-2B achieves the best mAP among end-to-end methods on the instance-level Inst2Seg benchmark, while remaining strong on semantic gIoU.The mAP–gIoU gap indicates that instance discrimination is harder than coarse semantic localization.
ReasonSeg: InstructSAM improves ReasonSeg cIoU over similarly sized X-SAM and SA2VA-4B by +5.0 on validation and +5.2 on the overall test set.The paper reports especially notable gains on long instructions, though the supplied passage is truncated before the associated figure.
Phrase-level referring segmentation: InstructSAM-2B surpasses GSVA-7B in gRefCOCO cIoU by +6.6 on val, +3.1 on testA, and +4.9 on testB, while also achieving higher multi-instance mAP.gRefCOCO includes multiple target instances, motivating the additional instance-level mAP metric.
Generalization benchmarks: InstructSAM achieves the best zero-shot GSEval performance, outperforming EVF-SAM by +1.5 gIoU, and exceeds RefTR-r50 by +12.9 on RoboRefIt testB.The RoboRefIt testB result is on a distribution-shifted split, highlighting generalization in indoor robotic perception.
Key designs: Removing learnable queries reduces Inst2Seg mAP from 31.5 to 20.1, while replacing hybrid with causal attention also degrades performance, especially on ReasonSeg.The query ablation causes consistent drops across all evaluated benchmarks.
Training and data ablations: Removing alignment drops Inst2Seg mAP from 31.5 to 8.1 and ReasonSeg validation cIoU from 65.0 to 15.9, while removing filtering lowers Inst2Seg mAP from 31.5 to 11.9.The filtering ablation also changes ReasonSeg cIoU/gIoU from 63.0/61.8 to 57.9/58.1, indicating label noise from unfiltered MLLM–SAM3 generation.

6 Conclusion

InstructSAM is a unified framework for instruction-driven multi-instance segmentation that translates arbitrary complex instructions into parallel instance slots and projects them into SAM3’s detector query space.

6 Conclusion: InstructSAM uses an explicit reasoning-to-instance query interface to convert arbitrary instructions into parallel, learnable instance slots for multi-instance segmentation.The framework supports free-form instructions involving attributes, relations, counting, exclusion, and implicit intent.

Appendix · A More Training Details

InstructSAM is trained in two stages: first aligning LLM-produced mask queries with SAM3 using large-scale grounding data, then fine-tuning reasoning-aware instruction segmentation. The appendix also reports multi-stage hyperparameters and training-dataset distributions.

A More Training Details: InstructSAM uses a two-stage training procedure comprising referring alignment pretraining followed by reasoning knowledge fine-tuning.Stage 2 starts from the aligned model obtained in Stage 1.
A More Training Details: Stage 1 pretrains learnable mask queries and explicitly aligns the LLM’s mask-query space with SAM3’s query space.The alignment uses category-level and phrase-level simple mask-grounding data.
A More Training Details: 2.5M training samples are used for Stage 1 alignment pretraining.These samples come from a large collection of category-level and phrase-level simple mask-grounding data.
A More Training Details: Stage 1 uses learning rates of 5 × 10^-6 for the MLLM and 1 × 10^-5 for the remaining trainable mask-decoder modules.The MLLM includes the LLM, vision encoder, and projector.
A More Training Details: The MLLM is initialized from Qwen3-VL-2B, while the mask decoder is initialized from pretrained SAM3.These initializations are specified for the multi-stage training setup.
A More Training Details: Stage 2 fine-tunes reasoning-aware segmentation knowledge to improve instruction following in complex referring and compositional scenarios.It combines instruction-level segmentation data with a subset of phrase-level referring data to preserve grounding robustness and improve reasoning generalization.

B More Visualization Results

The visualizations show that InstructSAM can follow complex instructions to localize and segment instances or regions. It handles compositional attributes, reasoning-based queries, and referring expressions involving appearance, position, and context.

Instruction-based Instance Segmentation: InstructSAM parses multi-attribute natural-language queries, distinguishes targets from similar distractors and clutter, and produces precise instance-level masks.The examples combine object categories, spatial relations, and contextual constraints.
Reasoning Segmentation: InstructSAM generates masks aligned with evidence regions inferred through multi-step reasoning beyond category recognition.Queries can require identifying functional parts or inferring intent and affordances.
Referring Segmentation: InstructSAM grounds referring expressions using appearance, relative position, or contextual relations and segments the corresponding regions.Examples include the far-right person in the background and bottles placed on a bench.

C Further Discussions

InstructSAM currently focuses on image inputs, leaving video segmentation as an important limitation because high-quality instruction-mask supervision is substantially harder to construct for videos.

Limitations: InstructSAM’s current version focuses on image inputs because large-scale, high-quality instruction-mask supervision is substantially more challenging for videos.Video annotation and automatic data generation must handle multi-instance interactions, temporal correspondence, and frame-level mask consistency.