A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future

1 INTRODUCTION

• Open-vocabulary detection and segmentation address the limits of small, predefined label sets that hinder real-world use.The cited examples include 20 Pascal VOC classes, 80 COCO classes, and 1,203 LVIS categories.Source: §1 · p. 1
• The survey groups methods by whether weak supervision is permitted and, when it is, which signals are used, yielding a taxonomy shared across tasks.It reviews detection, semantic, instance, and panoptic segmentation, as well as 3D scene and video understanding.Source: §1 · pp. 2–3
• Zero-shot methods are grouped into visual-semantic space mapping and novel visual feature synthesis, while open-vocabulary methods use four weak-supervision-based categories.The four categories are region-aware training, pseudo-labeling, knowledge distillation, and transfer learning.Source: §1 · pp. 2–3
• Region-aware training, pseudo-labeling, knowledge distillation, and transfer learning differ in their weak signals and use of VLM image encoders.Region-aware training uses image-text pairs; pseudo-labeling can use a VLM image encoder, while distillation and transfer learning mainly use one.Source: §1 · p. 3
• The taxonomy also applies to 3D scene and video understanding, where image-domain VLMs can bridge point clouds or videos and text.One example associates 2D OVD pseudo-labels with semantic labels via VLMs and back-projects them to 3D as supervision.Source: §1 · p. 3

2.1 Problem Definition

• OVD/OVS train on base-category annotations and test on both base and novel categories, while excluding annotated novel objects from training.The labeled categories are divided into disjoint base and novel sets.Source: §2.1 · p. 4

2.2 Related Domains and Tasks

• Related tasks differ in scope: visual grounding remains closed-set, weakly supervised detection uses image-level labels, open-set detection identifies one unknown class, and open-world detection adds newly learned classes over time.Open-world learning repeats this cycle without retraining from scratch at each step.Source: §2.2 · p. 4

2.3 Canonical Closed-Set Detectors and Segmentors

• Canonical detectors range from Faster R-CNN’s proposal-and-refinement stages to DETR’s set matching, while segmentation architectures include mask-based frameworks such as MaskFormer.Mask R-CNN adds a mask branch to Faster R-CNN; Mask2Former introduces masked cross-attention over predicted mask regions.Source: §2.3 · pp. 4–5

2.4 Large Vision-Language Models (VLMs)

• CLIP contrastively aligns image-caption pairs and uses prompted text embeddings as a classifier, while DINO and MAE support local image features for OVD/OVS.Text-guided diffusion models learn region-word correspondence, and parameter-efficient fine-tuning updates only a small set of additional parameters.Source: §2.4 · p. 5

3 ZERO-SHOT DETECTION (ZSD)

• Zero-shot detection transfers semantic knowledge from seen to unseen classes, using discriminative mapping or generative synthesis to address the scarcity of unseen visual data.Mapping separates decision boundaries in visual, semantic, or common spaces; synthesis generates unseen visual features for training.Source: §3 · p. 5

3.1.1 Learning a Mapping from Visual to Semantic Space

• Visual-to-semantic mapping projects RoI features into semantic space for classification, with background modeling techniques addressing ambiguity between background and unseen concepts.Bansal et al. use a max-margin loss and GloVe embeddings, alongside a fixed background label vector and WordNet classes assigned to background.Source: §3.1.1 · pp. 5–6

3.1.2 Learning a Joint Mapping of Visual-Semantic Space

• Joint-space methods combine information from visual and semantic spaces; MS-Zero averages similarity predictions from two unidirectional mappings.DPIF adds associative seen-class knowledge, while ContrastZSD contrasts seen region features with seen and unseen semantic embeddings.Source: §3.1.2 · p. 6

3.1.3 Learning a Mapping from Semantic to Visual Space

• Semantic-to-visual mapping embeds semantic features into visual space to address the hubness associated with projecting visual features into a lower-dimensional semantic space.Zhang et al. use a least-squares loss for this mapping.Source: §3.1.3 · p. 6

3.2 Novel Visual Feature Synthesis

• Novel visual feature synthesis trains a generator on seen semantic embeddings and visual features, then synthesizes unseen features to retrain a classifier while keeping the base model otherwise frozen.The synthesis uses unseen semantic embeddings and random noise to generate varied features.Source: §3.2 · p. 6, Figure 3 · p. 6
• Synthesis methods address feature consistency and context variation: DELO uses consistency losses, while GTNet adds foreground and background RoIs to target varied spatial context.RRFS uses intra-class semantic-diverging and inter-class structure-preserving losses; DELO's tightly bounded training objects otherwise limit context diversity.Source: §3.2 · pp. 6–7

4 ZERO-SHOT SEGMENTATION (ZSS)

• Zero-shot segmentation addresses semantic and instance segmentation at finer pixel-level granularity than zero-shot detection.Source: §4 · p. 7

4.1.1 Visual-Semantic Space Mapping

• Visual-semantic mapping methods classify pixels or prototypes in semantic space, with calibration and uncertainty-aware losses addressing prediction bias and uncertain samples.SPNet maps pixel features to semantic space; JoEm uses semantic prototypes and apollonius calibration, while PMOSR projects semantic embeddings to visual prototypes for flexible class addition at inference.Source: §4.1.1 · p. 7

4.1.2 Novel Visual Feature Synthesis

• Zero-shot segmentation synthesis methods use contextual structure or latent codes to generate more diverse visual features and adjust pseudo-label weighting by confidence.ZS3Net conditions synthesis on an adjacency graph; CaGNet uses contextual latent codes, and SIGN uses spatial latent codes and confidence-weighted pseudo annotations.Source: §4.1.2 · p. 7

4.2 Zero-Shot Instance Segmentation (ZSIS)

• Zero-shot instance segmentation maps visual features to semantic space and uses background-aware region proposals and a synchronized background strategy to distinguish background from unseen classes.Zheng et al. optimize the mapping with a mean-squared-error reconstruction loss.Source: §4.2 · p. 7

5 OPEN-VOCABULARY DETECTION (OVD)

• Open-vocabulary detection relaxes zero-shot detection's restriction against training images containing unannotated novel objects, using weak supervision signals.Source: §5 · p. 7

5.1 Region-Aware Training

• Region-aware training aligns proposals and words using image-text pairs, either through weakly supervised grounding or region-word matching losses.Ground-truth region-level descriptions can mitigate noise in image-level caption correspondence; GoldG combines region-level text datasets for grounding.Source: §5.1 · pp. 7–8

5.2 Pseudo-Labeling

• Pseudo-labeling methods combine base annotations with generated labels for known novel categories, varying the granularity from region-word pairs to region-caption pairs or captions.Methods address label quality through score fusion, iterative refinement, or alternative label generation strategies.Source: §5.2 · p. 9

5.3 Knowledge Distillation

• Knowledge distillation transfers visual knowledge from a vision-language teacher to a detector, with methods varying the region granularity and distillation objective.ViLD and ZSD-YOLO distill embeddings from base and novel proposals; other methods distill bags of regions, image regions, or embedding relationships.Source: §5.3 · pp. 9–10

5.4 Transfer Learning

• Transfer-learning methods use a pretrained vision-language image encoder during training and inference, either as a feature extractor or through fine-tuning, while some methods instead use its text encoder.Approaches include end-to-end fine-tuning, a frozen image encoder with a trained detection head, and unified detection and segmentation networks.Source: §5.4 · pp. 10–11

6.1.1 Region-Aware Training

• Region-aware methods improve image-text alignment through grounding, local-to-global learning, and region-level or sparse patch-word matching.Examples include filtering object-unrelated caption words, grounding masks to text, and matching grounded regions rather than whole images and captions.Source: §6.1.1 · p. 11

6.1.2 Pseudo-Labeling

• Zabari et al. refine category relevance maps with test-time augmentations, then use stochastic pixel sampling to create synthetic supervision.Source: §6.1.2 · p. 11

6.1.3 Knowledge Distillation

• Knowledge distillation transfers semantic relationships or combines complementary encoders, while ZeroSeg distills multi-scale CLIP features into segment tokens using unlabeled images.GKC enriches prompts with WordNet synonyms; SAM-CLIP merges SAM and CLIP encoders through cosine distillation.Source: §6.1.3 · p. 11

6.1.4 Transfer Learning

• Transfer learning spans text-encoder classification, frozen visual encoders, direct fine-tuning, and prompts or adapters for adapting visual features.The review distinguishes four strategies: use a text encoder as classifier, extract frozen visual features, fine-tune the visual encoder, or adapt it with prompts or lightweight adapters.Source: §6.1.4 · p. 11
• Frozen-feature approaches either ensemble a frozen VLM with a detector or use the VLM alone; other methods fine-tune encoders or adapt them with prompts and lightweight decoders.Examples include geometric ensembling for ZegFormer, local-feature adjustments in MaskCLIP, and SAN’s lightweight vision transformer on a frozen CLIP encoder.Source: §6.1.4 · pp. 12–13

6.2 Open-Vocabulary Instance Segmentation (OVIS)

• For instance segmentation, methods use caption-selected pseudo-regions, iterative pseudo-instances, diffusion-generated instances, or CLIP-SAM feature distillation.XPM down-weights noisy pseudo-labels by predicting their noise level; Mask-free OVIS avoids strong supervision on base categories and weak supervision on novel categories.Source: §6.2 · p. 13

6.3 Open-Vocabulary Panoptic Segmentation (OVPS)

• Panoptic methods use multi-scale matching, decoupled pixel and semantic queries, multi-dataset training, feature synthesis, and transfer strategies.APE trains with SA-1B using one head without distinguishing things from stuff; other approaches include prompt adaptation, mask-class tokens, and diffusion-derived mask features.Source: §6.3 · pp. 13–14

7.1 Open-Vocabulary 3D Scene Understanding

• Open-vocabulary 3D understanding is underexplored and data-scarce; methods commonly bridge point clouds and text through images and VLMs such as CLIP.Datasets typically provide projection matrices to transform point clouds into 2D boxes and vice versa.Source: §7.1 · p. 14

7.1.1 Open-Vocabulary 3D Object Detection (OV3D)

• 3D detection methods back-project 2D boxes or regions for localization and use CLIP-aligned image features for 3D recognition, while L3Det learns directly from 3D objects and text.CoDA progressively discovers novel 3D objects; the survey notes that point-cloud/image alignment may limit performance.Source: §7.1.1 · p. 14

7.1.2 Open-Vocabulary 3D Segmentation

• 3D segmentation methods align point features with CLIP or construct point-caption associations; instance methods aggregate multi-view features, cluster masks, or use scene-level images.PLA and RegionPLC also support instance segmentation; OpenIns3D avoids using images as the bridge between point clouds and text.Source: §7.1.2 · p. 14

7.2 Open-Vocabulary Video Understanding (OVVU)

• Video methods detect, segment, and track arbitrary instances, with BriVIS linking instance features across frames before aligning the bridge center to text embeddings.OV2Seg introduced the task and a 1,212-category benchmark; earlier methods align instances in separate frames without modeling their cross-frame correlation.Source: §7.2 · p. 14

8.1 Challenges

• Base-class bias appears in proposal quality, novel-to-base misclassification, and confidence; novel and background concepts are also difficult to distinguish.Proposed responses include frozen proposal generators, localization-based objectness, and inference-stage recalibration.Source: §8.1 · pp. 14–15
• Image-caption supervision has weak region-word correspondence, and pseudo-labels impose one-to-one matches but are usually generated once rather than iteratively refined.Source: §8.1 · p. 15
• VLM adaptation faces input-domain gaps and catastrophic forgetting; current detector architectures can be slow, while lightweight models may reduce novel-object recall.The survey also questions whether small models can absorb large-VLM knowledge and notes that synonym predictions may not count as true positives under current metrics.Source: §8.1 · p. 15

8.2 Future Directions

• Future directions include extending open-vocabulary methods to more tasks, unifying detection and segmentation, using multimodal LLMs, and combining foundation models.The survey describes SAM, CLIP, DINO, and text-to-image diffusion models as having different capabilities relevant to such combinations.Source: §8.2 · p. 15
• Real-time OVD and OVS remain a research direction because current models have heavy backbones and necks unsuitable for real-time applications.Source: §8.2 · p. 15

9 CONCLUSION

• The survey reviews nearly two hundred OVD and OVS methods across detection, segmentation, 3D scene understanding, and video understanding.Source: §9 · p. 16
• Its methodology taxonomy groups methods by weak-supervision permissions and usage into six categories intended to apply across tasks.Source: §9 · p. 16
• The survey benchmarks state-of-the-art methods and vital components for each task in the appendix, alongside its discussion of challenges and promising directions.Source: §9 · p. 16

• Table 2 summarizes datasets and evaluation metrics.Source: Table 2 · p. 21
• The supplementary material provides detailed, comprehensive comparisons of methods across different tasks and settings.Source: APPENDIX · p. 21
• The benchmark does not distinguish backbone initialization, background evaluation, or validation-set-version differences; consult the original papers for precise details.Source: APPENDIX · p. 21

• Generalized evaluation tests both base and novel classes, while cross-dataset transfer evaluation tests a source-trained model on target datasets without adaptation.The survey describes generalized evaluation as more challenging and realistic; open-vocabulary detection and segmentation mainly use it and also use cross-dataset transfer evaluation.Source: Evaluation Protocols, Metrics, and… · p. 21
• Detection and instance segmentation use box or mask AP at fixed IoU thresholds or integrated over IoU thresholds from 0.5 to 0.95.AP may be reported separately for base and novel classes; LVIS rare-category AP is APr, and recall is complementary.Source: Evaluation Protocols, Metrics, and… · p. 21
• Semantic segmentation uses base- or novel-class mIoU and their harmonic mean, while panoptic segmentation uses PQ, viewed as SQ multiplied by RQ.For zero-shot detection, AP50 may instead denote the harmonic mean of novel- and base-class AP50; the survey does not differentiate these usages.Source: Evaluation Protocols, Metrics, and… · p. 21
• Zero-shot detection tables cover COCO, Pascal VOC, ILSVRC-2017 detection, and Visual Genome; the zero-shot semantic segmentation table covers Pascal VOC and Pascal Context.The Visual Genome and ILSVRC-2017 table uses non-generalized evaluation and reports Recall@100; the semantic segmentation table shows results for four novel classes.Source: Table 3 · p. 21, Table 4 · p. 21, Table 5 · p. 22
• Additional tables cover open-vocabulary 3D detection on SUN RGB-D, ScanNet, and nuScenes; 3D instance segmentation on ScanNet200; and video instance segmentation on four datasets.The video datasets are Youtube-VIS19, Youtube-VIS21, BURST, and LV-VIS.Source: Table 7 · p. 22, Table 8 · p. 22, Table 9 · p. 22
• Open-vocabulary detection tables cover generalized COCO and LVIS evaluation and cross-dataset transfer across Pascal VOC, Objects365, COCO, OpenImages, and LVIS.Source: Table 10 · p. 23, Table 11 · p. 24, Table 12 · p. 25
• Open-vocabulary semantic segmentation tables report generalized evaluation and cross-dataset transfer results across ADE20K, Pascal Context, Pascal VOC, Cityscapes, and COCO datasets.Source: Table 13 · p. 26, Table 14 · p. 26
• The tables include open-vocabulary instance segmentation on COCO and OpenImages, and panoptic segmentation on COCO Panoptic and ADE20K, including cross-dataset transfer results.The panoptic table identifies PQ for stuff and thing classes as PQst and PQth.Source: Table 15 · p. 27, Table 16 · p. 27, Table 17 · p. 27