PaperlayerView exact revision on arXiv

Source-linked AI summary

Tokenize Anything via Prompting

Ting Pan, Lulu Tang, Xinlong Wang, Shiguang Shan

arXiv:2312.09128v2cs.CV

TL;DR

Existing models often specialize in either class-agnostic localization or visual semantics, and a unified promptable model also lacks aligned web-scale data. TAP combines segmentation masks with CLIP-derived semantic priors in a promptable tokenizer, reporting results across recognition, segmentation, and captioning.

  • Problem

    Existing models often focus on class-agnostic masks or visual semantics, while promptable generalist perception and web-scale data pairing masks, classes, and captions remain unavailable.

    Source: §1 · p. 2

  • Method

    TAP adds a semantic token to each predicted mask and jointly trains promptable segmentation and concept prediction using SA-1B masks with CLIP-derived priors.

    Source: §1 · p. 2, §3.1 · pp. 6–7

  • Results

    TAP reports 59.1 AP on LVIS, 43.0 vs. 43.1 AP for TAP and SAM, and a CIDEr score of 164.7 on Visual Genome.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present TAP as a versatile region-level image tokenizer for encoding general-purpose region context across visual perception tasks.

    Source: §5 · p. 14

    TAP uses a human-curated label space that falls short of an open-world assumption, and its text decoder is fine-tuned on constrained region-caption data.

    Source: F Limitations · p. 23

Abstract

from arXiv · show

We present a unified, promptable model capable of simultaneously segmenting, recognizing, and captioning anything. Unlike SAM, we aim to build a versatile region representation in the wild via visual prompting. To achieve this, we train a generalizable model with massive segmentation masks, \eg, SA-1B masks, and semantic priors from a pre-trained CLIP model with 5 billion parameters. Specifically, we construct a promptable image decoder by adding a semantic token to each mask token. The semantic token is responsible for learning the semantic priors in a predefined concept space. Through joint optimization of segmentation on mask tokens and concept prediction on semantic tokens, our model exhibits strong regional recognition and localization capabilities. For example, an additional 38M-parameter causal text decoder trained from scratch sets a new record with a CIDEr score of 164.7 on the Visual Genome region captioning task. We believe this model can be a versatile region-level image tokenizer, capable of encoding general-purpose region context for a broad range of visual perception tasks. Code and models are available at {\footnotesize \url{https://github.com/baaivision/tokenize-anything}}.

1 Introduction

The introduction frames a need for one promptable model that combines localization and regional semantics, and presents TAP as a framework, dataset, and training method for doing so. Its reported results span recognition, segmentation, and region captioning.

1 Introduction

  • The introduction identifies a gap between models focused on class-agnostic masks and models focused on visual semantics, motivating a unified regional model.It describes localization and recognition as complementary requirements for understanding arbitrary regions.Source: §1 · p. 2
  • TAP adds a semantic token to each predicted mask, pairing pixel-wise segmentation with region-level semantic prediction and captioning from flexible prompts.The introduction describes an MLP head for open-vocabulary classification and a lightweight text decoder for promptable captioning.Source: §1 · p. 2, Figure 2 · p. 3
  • SemanticSA-1B combines SA-1B segmentation data with CLIP-predicted concept distributions for segmented regions, addressing the lack of web-scale aligned data.The concept distributions come from a CLIP model trained on LAION-2B image-text pairs.Source: §1 · pp. 2–3
  • 59.1 AP on LVIS, 43.0 vs. 43.1 AP for TAP and SAM, and a CIDEr score of 164.7 on Visual Genome demonstrate the reported recognition, segmentation, and captioning results.The paper reports 59.1 AP for zero-shot instance classification, comparable zero-shot segmentation performance, and a new Visual Genome region-captioning record.Source: §1 · p. 3

2 Related work

Related work covers vision foundation models, open-vocabulary segmentation, and region understanding. TAP is positioned among approaches that combine prompting, segmentation, and region-level semantics.

2.1 Vision Foundation Models

  • Vision foundation models include CLIP for image-text alignment, SAM for promptable segmentation, and SegGPT for in-context segmentation across varied tasks.The section also mentions generalist models built using multimodal datasets.Source: §2.1 · p. 4

2.2 Open-Vocabulary Segmentation

  • Open-vocabulary segmentation work integrates VLM knowledge through pixel-text alignment, two-stage mask classification, weak caption supervision, and joint-task approaches.TAP is distinguished from two-stage approaches that use image-level CLIP to classify masks because it focuses on region-level semantic awareness.Source: §2.2 · p. 4

2.3 Zero-shot Region Understanding

  • Region-understanding approaches include SAM-CLIP and RegionSpot, while SEEM and ASM target broader prompted or panoptic recognition; TAP instead uses SA-1B masks with CLIP priors.The section contrasts TAP’s use of segmentation masks and semantic priors with models relying on handcrafted multimodal datasets.Source: §2.3 · p. 5

3 Approach

TAP builds a promptable tokenizer for arbitrary regions by combining segmentation masks with CLIP-derived semantics, then extends it to generate captions. Its inference pipeline uses the selected semantic token for recognition and caption generation.

3 Approach

  • TAP pre-trains a promptable tokenizer on segmentation masks with CLIP priors, then extends it with generative promptable captioning.The framework targets segmentation, recognition, and captioning of arbitrary regions of interest.Source: §3 · p. 5

3.1 Promptable Tokenization

  • The decoder adds a semantic token to each predicted mask: the mask token supports pixel-wise segmentation, while the semantic token supports region-level recognition.The model retains SAM’s architecture with an image encoder, prompt encoder, and image decoder.Source: §3.1 · p. 6
  • SemanticSA-1B pairs SA-1B image segments with EVA-CLIP concept distributions over a 2,560-category label list.The CLIP-derived priors are computed for each masked segment and stored with the segmentation data.Source: §3.1 · p. 6
  • Concept alignment minimizes KL divergence between predicted and target concept distributions, encouraging predicted visual embeddings toward relevant concepts and away from irrelevant ones.The paper contrasts this distribution-level alignment with feature alignment and describes the resulting embeddings as orthogonal.Source: §3.1 · p. 7
  • Pre-training jointly optimizes concept prediction and promptable segmentation with equal loss weights, using a combined objective on SemanticSA-1B.The segmentation setup samples box or point prompts, then samples points from mask-error regions; it also includes noninteractive mask-based prompt sampling.Source: §3.1 · p. 7

3.2 Promptable Captioning

  • A lightweight causal text decoder uses semantic tokens as context to generate region captions, avoiding the need for an LLM in this setup.The decoder is trained end-to-end with causal language modeling; its semantic token is placed before the [BOS] and word tokens.Source: §3.2 · p. 8

3.3 Inference

  • At inference, the decoder selects a mask and associated semantic token from four masks, then uses that token for concept prediction and caption generation.Caption generation produces up to 40 word tokens with greedy sampling.Source: §3.3 · p. 9

4 Experiments

The experiments evaluate TAP on zero-shot classification, segmentation, and region captioning, then examine its training choices and qualitative behavior. TAP achieves a Visual Genome captioning record while retaining segmentation performance comparable to SAM.

4.1 Experiments Setup

  • SemanticSA-1B combines 1.1 billion SA-1B masks from 11 million images with CLIP priors over 2,560 categories.Evaluation includes zero-shot classification and segmentation plus region captioning; captioning uses a two-stage fine-tuning setup.Source: §4.1 · p. 9

4.2 Main Results

  • TAP substantially surpasses RegionCLIP and RegionSpot in zero-shot LVIS classification, while EVA-CLIP performs best but requires impractical massive-crop computation.The evaluation uses ground-truth boxes, and LVIS covers 1,203 categories.Source: §4.2 · p. 10, Table 1 · p. 10
  • 164.7 CIDEr on Visual Genome sets a new region-captioning record with full fine-tuning and a lightweight text decoder.The reported partial-FT score is 154.7 CIDEr.Source: §4.2 · p. 11, Table 2 · p. 11
  • TAP achieves comparable zero-shot instance-segmentation results to original SAM across model scales on COCO and LVIS.The comparison uses boxes from ViTDet-H; the authors report that adding concept prediction and region captioning does not compromise SAM’s segmentation capability.Source: §4.2 · p. 11, Table 3 · p. 12

4.3 Ablation Study

  • Concept prediction improves captioning over segmentation-only pre-training, and outperforms feature prediction for classification and captioning without changing mask AP.The authors identify the concept space as important for acquiring CLIP priors.Source: §4.3 · p. 12, Table 4 · p. 12, Table 5 · p. 12
  • Semantic tokens outperform mask tokens for captioning while maintaining comparable segmentation AP.The default model jointly trains promptable segmentation and concept prediction, then uses semantic tokens to prompt the text decoder.Source: Table 4 · p. 12, §4.3 · p. 13
  • Increasing text-decoder scale yields no substantial Visual Genome improvement unless text length or quantity also increases.This ablation concerns the captioning bottleneck.Source: §4.3 · p. 13, Table 6 · p. 13

4.4 Qualitative Results

  • Point prompts can produce masks, category names, and descriptions; qualitative examples include open-world concepts and accurate segmentation of crowded scenes.The paper notes that curated concepts such as brand names and fictional characters can be difficult to retrieve through classification.Source: §4.4 · pp. 13–14, Figure 5 · p. 13, Figure 6 · p. 14

5 Conclusion

The conclusion presents TAP as a promptable model for jointly segmenting, recognizing, and captioning arbitrary regions. It identifies semantic priors, promptable tokenization, and concept prediction as components of this approach.

5 Conclusion

  • TAP combines semantic priors injected into SA-1B, promptable tokenization, and concept prediction to support joint regional perception.The conclusion frames visual prompting as extending beyond segmentation and reports that SAM can gain regional semantic awareness without compromising mask AP.Source: §5 · p. 14
  • The authors identify an orthogonal vocabulary concept space as essential for effectively learning CLIP priors.They also state that visual prompting can support a broader range of tasks than segmentation alone.Source: §5 · p. 14
  • The paper aims to advance from segmenting anything toward perceiving anything via prompting.The authors hope the work will inspire more compact and significant vision foundation models.Source: §5 · p. 14

Appendix

  • The appendix covers multimodal preprocessing, caption fine-tuning, point-prompt evaluation, additional visualizations and comparisons, and limitations.Source: Appendix · p. 20

A Pre-processing Details

The preprocessing details describe how SA-1B region crops and a curated concept vocabulary are represented with EVA-CLIP embeddings. These visual and text embeddings support concept-distribution construction.

A Pre-processing Details

  • Each of SA-1B’s 1.1 billion masked segments is cropped, resized onto a 224×224 canvas, and encoded by EVA-CLIP as a 1,024-dimensional visual embedding.The embeddings require approximately 2.25 TB of storage, alongside an SA-1B database of around 10.55 TB.Source: A Pre-processing Details · p. 20
  • Figure 7 presents the pipeline for constructing SemanticSA-1B.Source: Figure 7 · p. 20
  • A curated set of 2,560 concepts is converted to CLIP text embeddings using a template such as “a {}”.The text embeddings are used to compute predicted or target concept distributions.Source: A Pre-processing Details · p. 20

B Caption Fine-tuning Details

The caption fine-tuning details distinguish partial fine-tuning from end-to-end training. The latter uses generated masks and CLIP priors for Visual Genome regions and improves caption performance.

B Caption Fine-tuning Details

  • Partial fine-tuning freezes the image encoder-decoder and trains only the text decoder on Visual Genome captions.This setting is intended to avoid learning specific segmentation biases from limited region-caption data and does not train promptable captioning to contribute to segmentation.Source: B Caption Fine-tuning Details · p. 20
  • Full fine-tuning generates Visual Genome masks and EVA-CLIP priors, then jointly enables localization, recognition, and captioning.The reported CIDEr score rises from 154.7 with partial fine-tuning to 164.7 with full fine-tuning.Source: B Caption Fine-tuning Details · p. 21
  • The authors propose exploring TAP as a data engine for generating regional vision-language annotations in future work.They describe this potential as an asset for training large vision-language models.Source: B Caption Fine-tuning Details · p. 21

C Evaluation with point prompts

Point prompts sampled from a ground-truth mask were evaluated against box prompts for classification on LVIS. Five points performed comparably to a box prompt.

C Evaluation with point prompts

D More Visualizations

The visualizations show TAP segmenting, recognizing, and captioning objects with various prompts, including in artistic images; failure cases are also visualized.

D More Visualizations

E Comparison with SEEM and SAM-CLIP

TAP is compared with SEEM and SAM-CLIP on class-specific instance segmentation, with prompting and task settings specified for the comparison.

E Comparison with SEEM and SAM-CLIP

  • TAP marginally outperforms SEEM in class-specific instance segmentation when prompted with ViTDet boxes, despite lacking object queries.The comparison uses predicted masks from queries and categories from CLIP text embeddings.Source: E Comparison with SEEM and SAM-CLIP · pp. 22–23
  • TAP naturally performs class-specific instance segmentation and achieves better results than SAM-CLIP, whose two task heads are not straightforward to integrate for that task.Source: E Comparison with SEEM and SAM-CLIP · p. 23
  • The table covers instance segmentation with ViTDet box prompts and semantic segmentation with 16 × 16 grid point prompts.Source: Table 7 · p. 23

F Limitations

TAP is constrained by its human-curated label space and limited region-caption training data, which affect concept ranking and broader language understanding.

F Limitations

  • The human-curated label space falls short of an open-world assumption and leads to unstable ranking of similar concepts at inference.Source: F Limitations · p. 23
  • The text decoder's constrained region-caption training data may limit scalability and breadth of vision-language understanding.Source: F Limitations · p. 23
  • Simple quantity annotations do not solve object counting well; expanding caption data is expected to instruct models for complex understanding.Source: F Limitations · p. 23
Loading 2312.09128v2…