Source-linked AI summary
SAMTok: Representing Any Mask with Two Words
Yikang Zhou, Tao Zhang, Dengxian Gong, Yuanzheng Wu, Ye Tian, Haochen Wang, Haobo Yuan, Jiacong Wang, Lu Qi, Hao Fei, Anran Wang, Zhuochen Wang, Yujing Wang, Cheng Chen, Shunping Ji, Xiangtai Li
TL;DR
Pixel-wise MLLMs are difficult to scale because they rely on complex region encoders, segmentation decoders, and incompatible training objectives. SAMTok converts masks into two textual special tokens so QwenVL models can learn mask understanding and generation with standard language-model training and text-based reinforcement learning. Across diverse benchmarks, QwenVL-SAMTok achieves strong or comparable performance, while textual answer-matching rewards produce additional gains on GRES and GCG.
Problem
Pixel-wise MLLMs lack a simple scalable design combining mask understanding, mask generation, and compatible training objectives.
Method
SAMTok converts region masks into two discrete textual tokens and enables MLLMs to model them through next-token prediction and textual answer-matching reinforcement learning.
Results
QwenVL-SAMTok achieves stronger or comparable performance across diverse region-level benchmarks, while GRPO improves GRES and GCG metrics and surpasses previous SOTA results.
Takeaways & Limitations
SAMTok supports a unified, language-native paradigm for equipping MLLMs with pixel-wise understanding and generation capabilities.
Takeaways & Limitations
SAMTok reconstructs 2D image region masks but does not yet reconstruct video masks or represent points, lines, and boxes.
Abstract
from arXiv · showhide
Pixel-wise capabilities are essential for building interactive intelligent systems. However, pixel-wise multi-modal LLMs (MLLMs) remain difficult to scale due to complex region-level encoders, specialized segmentation decoders, and incompatible training objectives. To address these challenges, we present SAMTok, a discrete mask tokenizer that converts any region mask into two special tokens and reconstructs the mask using these tokens with high fidelity. By treating masks as new language tokens, SAMTok enables base MLLMs (such as the QwenVL series) to learn pixel-wise capabilities through standard next-token prediction and simple reinforcement learning, without architectural modifications and specialized loss design. SAMTok builds on SAM2 and is trained on 209M diverse masks using a mask encoder and residual vector quantizer to produce discrete, compact, and information-rich tokens. With 5M SAMTok-formatted mask understanding and generation data samples, QwenVL-SAMTok attains state-of-the-art or comparable results on region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and multi-round interactive segmentation. We further introduce a textual answer-matching reward that enables efficient reinforcement learning for mask generation, delivering substantial improvements on GRES and GCG benchmarks. Our results demonstrate a scalable and straightforward paradigm for equipping MLLMs with strong pixel-wise capabilities. Our code and models are available.
1 Introduction
SAMTok addresses scalability challenges in pixel-wise MLLMs by representing masks as compact textual tokens that support unified understanding, generation, and reinforcement learning. QwenVL-SAMTok achieves strong or comparable performance across diverse region-level tasks, with further gains from textual answer-matching rewards.
- Challenges: Pixel-wise MLLMs face fragmented mask-input and mask-output designs, incompatible training pipelines, and costly representations containing dozens or hundreds of tokens.Existing systems use complex region encoders, specialized segmentation decoders, additional modules, and heterogeneous losses or forward pipelines.
- Approach: SAMTok tokenizes diverse region masks into textual special tokens and reconstructs them, creating a language-like interface without architectural modifications or specialized loss design.The tokenizer is initialized from SAM2 and uses a mask encoder with residual vector quantization to produce two discrete tokens.
- Training: Approximately 5M mask-understanding and generation samples are converted into standard image-text VQA data for supervised fine-tuning of QwenVL models.This formulation lets MLLMs learn pixel-wise capabilities through standard next-token prediction.
- Results: QwenVL-SAMTok matches or surpasses expert MLLMs across region captioning, region VQA, grounded conversation, referring segmentation, scene graph parsing, and interactive segmentation.The reported evaluation spans mask understanding, mask generation, and interleaved understanding-generation tasks.
- Results: After GRPO, textual answer-matching rewards improve GRES validation by 8.9% in gIoU and 21.0% in N-acc, and GCG validation by 4.7% in AP50 and 6.6% in Recall.These results exceed previous SOTA methods by 4.3% in gIoU, 8.3% in N-acc, 8.3% in AP50, and 8.4% in Recall.
2 Method
SAMTok represents region masks as compact discrete tokens, allowing MLLMs to process mask understanding and generation through a unified text-based interface. Its SAM-based tokenizer and decoder support reconstruction, multi-task training, and reinforcement learning for mask generation.
- SAMTok: SAMTok combines a mask VAE with vector quantization to convert region masks into compact discrete representations.Its encoder and decoder use SAM components, while vector quantization discretizes continuous mask embeddings.
- SAMTok: The encoder maps an image and region mask to a continuous embedding, then residual quantization produces two discrete codebook tokens.Residual quantization performs successive nearest-neighbor lookups using the embedding and its residual.
- SAMTok: The decoder treats the two discrete embeddings as language prompt embeddings and reconstructs the corresponding 2D mask for the image.The discrete embeddings form a sparse prompt that interacts with image features inside SAM’s decoder.
- Unified mask-token interface: SAMTok converts mask understanding and generation into a unified textual interface by mapping masks to special words and predicted words back to segmentation masks.This supports mask-to-text, text-to-mask, interleaved, and interactive tasks in a common formulation.
- Reinforcement Learning for Mask Generation: SAMTok’s discrete representations make mask generation compatible with reinforcement learning, including rewards based on matching predicted mask words to ground-truth answers.The reward counts true-positive mask words relative to predicted masks while penalizing repetitive predictions through the denominator definition.
3 Experiment
SAMTok is evaluated through a three-stage training pipeline and diverse pixel-wise tasks, using large mask and conversation datasets. QwenVL-SAMTok achieves strong or state-of-the-art results across text–mask generation, interactive segmentation, grounding, and mask-to-text benchmarks.
- Experiment Setup: The training process comprises SAMTok training, MLLM supervised fine-tuning, and reinforcement learning.The datasets include 209M masks for tokenizer training and approximately 5M supervised fine-tuning samples.
- Experiment Setup: The supervised dataset covers mask generation, region understanding, and collaborative multi-turn mask generation and understanding tasks.Examples include grounding, referring segmentation, grounded conversation, scene parsing, region captioning, region VQA, and interactive reasoning segmentation.
- Main Results: SAMTok achieves state-of-the-art GCG performance, improving validation captioning by +1.3% METEOR and +5.5% CIDEr and mask metrics by +5.3% AP50, +5.2% mIoU, and +4.7% Recall.The test set shows similar gains across both textual and visual metrics.
- Main Results: On multi-round interactive segmentation, SAMTok surpasses previous best results by an average of +7.7% on MR-RefCOCO/+/g and +10.7% on MR-PACO.The benchmarks test object-level and part-level reasoning across multiple interaction rounds.
- Main Results: For text-to-mask tasks, the model improves average gIoU by +1.5% and average N-acc by +4.3% across GRES splits, while achieving 67.8 versus 62.6 in zero-shot GroundingSuite gIoU.It also achieves a new SOTA among models with fewer than 4B parameters on RefCOCO/+/g.
- Main Results: The model remains competitive on mask-to-text and grounding tasks without architectural modifications, including DLC-Bench region captioning and RefCOCO-family visual grounding.On DLC-Bench, it scores 65.6 versus DAM’s 67.3 and exceeds general Qwen2.5VL-7B’s 41.2.
4 Related Work
Related work addresses region understanding and generation through visual prompts, ROI features, textual coordinates, special tokens, or autoregressively generated mask images. SAMTok instead represents regions with two special text tokens and supports text-only reward matching for mask generation.
- Existing region-understanding methods use visual prompts, coordinates, or ROI features, but specialized designs and pipelines can limit precision, generalizability, and scalability.
- SAMTok compresses input regions into two special text tokens, preserving precise region representation without affecting image content or coupling to the MLLM.
- Mask-generation approaches include textual points, boxes, or polygons; special tokens with dedicated decoders; and autoregressively generated mask images.
- A textual answer-matching reward checks whether predicted answers contain target region tokens, avoiding detokenization, external tools, and auxiliary correctness models.
5 Conclusion
SAMTok presents a unified textual interface for representing region masks and training pixel-wise multimodal models. The paper concludes that this design supports scalable pixel-wise reasoning while remaining limited to 2D masks.
- SAMTok represents any region mask using two discrete tokens, allowing MLLMs to generate and understand masks through a unified textual interface.
- The approach reformulates mask understanding and generation as standard next-token prediction without task-specific architectures or loss functions.
- SAMTok demonstrates enhanced pixel-wise understanding and generation across diverse benchmarks and supports text-based reward optimization for mask generation.
- SAMTok currently reconstructs only 2D image region masks, leaving video masks and entities such as points, lines, and boxes for future work.
A Overview
The overview appendix organizes supplementary material around implementation details, experiments, ablations, and visualizations. It directs readers to sections B through E for these materials.
- Section B provides additional implementation details for the tokenizer and multimodal large language models.
- Section C contains additional experimental results.
- Section D presents ablation studies of tokenizer designs and their effectiveness.
- Section E provides additional visualizations.
B Implementation Details
SAMTok uses pretrained SAM 2.1 components, discrete mask-token codebooks, and Qwen-VL models with limited vocabulary modifications. Training separately specifies tokenizer, supervised fine-tuning, and reinforcement-learning settings.
- The SAMTok encoder and decoder initialize from pretrained SAM 2.1 Large weights, while the quantizer codebook is randomly initialized.
- During tokenizer training, SAM image and prompt encoders are frozen, the SAM decoder is trainable, and quantization uses two non-shared codebooks of size 256.
- The base Qwen-VL models receive 512 mask tokens plus start and end tokens as their only architectural vocabulary modification.
- During supervised fine-tuning and reinforcement learning, the MLLM image encoder is frozen while the projection layer and language model are fine-tuned.
C Additional model experiments
Additional experiments show that SAMTok generalizes across panoptic scene graph generation and different MLLM architectures without task-specific designs.
- Panoptic scene graph generation: SAMTok converts 2D masks into specialized word tokens for task generalization without task-specific designs.
- SAMTok integration across MLLMs: The decoupled SAMTok interface is evaluated with both tile-based and adaptive-resolution MLLMs using the same training data.The evaluated examples include PerceptionLM and the Qwen-VL series.
- SAMTok integration across MLLMs: The mask-token interface works effectively across diverse MLLMs.
D Ablation Study
The ablations evaluate reconstruction and generation using r-Acc and g-Acc, showing that quantization choices and residual-step counts materially affect fidelity.
- Set Up: SAMTok is evaluated with mask IoU for reconstruction and mean cIoU across RefCOCO, RefCOCO+, and RefCOCOg for generation.The reconstruction evaluation uses 23,754 EntitySeg validation masks and Qwen2.5-VL-3B as the base model for generation.
- Quantization: Reducing standard VQ’s codebook from 65,536 to 1,024 significantly lowers reconstruction and generation accuracy.With a large codebook, FSQ improves codebook utilization and outperforms standard VQ on both measures.
- Codebook size and quantization steps: 0.75 vs. 0.70 in r-Acc: using 1024×4 instead of 1024×2 residual quantization steps improves reconstruction fidelity.The larger configuration expands the search space from 1024^2 to 1024^4, making MLLM learning more difficult.
E Visualization
Visualizations illustrate SAMTok’s mask reconstruction quality and its use for grounded descriptions, referring segmentation, relational scene understanding, and interleaved text–mask generation.
- Notation: Mask words are displayed as quant-code pairs such as <11-347>.
- SFT vs. RL: RL improves target recall, relative-position localization, and mask quality over SFT on GRES examples.
- PSG visualizations: SAMTok supports dense panoptic scene understanding by jointly generating subject–relation–object descriptions and pixel-aligned masks.
- GRES visualizations: SAMTok grounds attribute-dependent, context-dependent, and part-level referring expressions, including scenes with occlusion or clutter.
- Region caption visualizations: Two special tokens enable detailed, contextually relevant region captions while reducing ambiguity relative to bounding-box grounding.
- GCG visualizations: SAMTok links phrases to precise pixel regions while jointly generating narrative captions and aligned masks.
- Mask reconstruction: Two-token mask codes preserve fine structures for small, thin, and irregular objects under challenging lighting or clutter.Reconstruction uses the original image and quantized mask tokens, while remaining decoupled from downstream MLLM training.