PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open Vocabulary Learning: A Survey

Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, Dacheng Tao

arXiv:2306.15880v4cs.CVcs.AI

TL;DR

Visual scene models are often limited to training-set categories, although real scenes can contain more categories than datasets annotate. This survey reviews open-vocabulary learning, categorizes more than 50 models, and compares detection and segmentation results. It synthesizes challenges and future research directions.

  • Problem

    Conventional scene-understanding approaches use close-set categories, while real-world scenes may contain more categories than training datasets annotate.

    Source: §1 · p. 1

  • Method

    The survey reviews related settings, categorizes more than 50 models by task and technical viewpoint, and compares detection and segmentation approaches and results.

    Source: §5 · p. 15

  • Results

    The survey provides fair comparisons under stated settings and synthesizes challenges and future research directions.

    Source: §5 · p. 15

  • Takeaways & Limitations

    For video open-vocabulary detection and tracking, the survey identifies a need for more dynamic, challenging datasets to explore vision-language models’ potential.

    Source: §4.2 · p. 14

    Video open-vocabulary detection and tracking remain sparsely studied, and the few existing examples use simple scenes.

    Source: §4.2 · p. 14

Abstract

from arXiv · show

In the field of visual scene understanding, deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. However, most approaches operate on the close-set assumption, meaning that the model can only identify pre-defined categories that are present in the training set. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. These new approaches seek to locate and recognize categories beyond the annotated label space. The open vocabulary approach is more general, practical, and effective compared to weakly supervised and zero-shot settings. This paper provides a thorough review of open vocabulary learning, summarizing and analyzing recent developments in the field. In particular, we begin by comparing it to related concepts such as zero-shot learning, open-set recognition, and out-of-distribution detection. Then, we review several closely related tasks in the case of segmentation and detection, including long-tail problems, few-shot, and zero-shot settings. For the method survey, we first present the basic knowledge of detection and segmentation in close-set as the preliminary knowledge. Next, we examine various scenarios in which open vocabulary learning is used, identifying common design elements and core ideas. Then, we compare the recent detection and segmentation approaches in commonly used datasets and benchmarks. Finally, we conclude with insights, issues, and discussions regarding future research directions. To our knowledge, this is the first comprehensive literature review of open vocabulary learning. We keep tracing related works at https://github.com/jianzongwu/Awesome-Open-Vocabulary.

1 INTRODUCTION

Conventional scene-understanding models are constrained by fixed training categories and costly annotations, while open vocabulary learning uses language and vision-language knowledge to extend recognition to novel classes. This survey reviews the field’s concepts, methods, domains, and benchmarks.

Introduction

  • Fixed category sets and costly object annotations make conventional models difficult to extend to real scenes with more concepts than their training data covers.The introduction notes that COCO contains 80 categories, while natural images often contain more, making broader detector coverage expensive to annotate.Source: §1 · p. 1
  • Unlike zero-shot learning, open vocabulary learning can use visual-related language data as auxiliary supervision to recognize novel classes beyond the base categories.Captions are less costly than box or mask labels and may include novel class names, attributes, and object motions.Source: §1 · pp. 1–2
  • Visual-language models align images and vocabulary representations, helping open vocabulary methods extend recognition to new domains without collecting additional labeled data.The passage attributes this practical potential to VLM pretraining on large-scale image-text pairs.Source: §1 · p. 2
  • The survey systematically covers open vocabulary learning across detection, segmentation, video, and 3D scene understanding, including related concepts, representative methods, settings, and benchmarks.It presents itself as the first survey focused specifically on this area.Source: §1 · p. 2

2 BACKGROUND

Section 2 defines open vocabulary learning, traces its development, summarizes relevant tasks, datasets, and metrics, and reviews related research domains. It contrasts learning to recognize novel classes using broad language knowledge with approaches that reject, detect, or incrementally learn unknowns.

2.1 Concept Definition

  • Open vocabulary learning augments base-class training pairs with vision-aware language data and evaluates predictions over both base and novel classes.The language vocabulary need not cover either label set and may include additional words that further extend generalizability.Source: §2.1 · p. 3
  • Open-set learning rejects unknowns, open-world learning incrementally adds labeled categories, OOD detection identifies distribution shifts, and zero-shot learning classifies unseen classes.Open-set methods need not classify unknown categories further, whereas zero-shot methods distinguish among novel classes.Source: §2.1 · pp. 3–4

2.2 History and Roadmap

  • Open-vocabulary work grew from joint pixel-word embeddings through OVR-CNN, ViLD, and LSeg, alongside expanding research and benchmarks.SAM [42] combined with CLIP supports zero-shot segmentation without fine-tuning; the survey notes increased research since 2021 and growing language knowledge from LLMs [43].Source: §2.2 · p. 4

2.3 Tasks, Datasets, and Metrics

  • The survey focuses on detection, instance and semantic segmentation, and tracking, while also considering attribution prediction, video classification, and point-cloud classification.It lists COCO and LVIS for detection, v3Det with more than 10,000 categories, and datasets including ADE20k, PASCAL-VOC 2012, PASCAL-Context, Cityscapes, VSPW, Youtube-VIS, LV-VIS, MOSE, and TAO. Metrics include mAP and mAR for detection, mIoU for semantic segmentation, mask-based mAP for instance segmentation, and PQ for panoptic segmentation.Source: §2.3 · p. 4

2.4 Related Research Domains

  • Related domains include open-set recognition, open-world learning, and OOD detection, which respectively reject unknowns, incrementally learn labeled categories, and identify distribution-shifted samples.Open-set approaches include discriminative boundary or margin methods and generative methods; OOD approaches include classification-, density-, distance-, and reconstruction-based methods.Source: §2.4 · p. 4
  • Zero-shot detection and segmentation use discriminative or generative methods, but pixel-level formulations can be poorly matched to text embeddings describing objects or segments.SPNet projects pixel features using fixed semantic embeddings, while ZS3Net generates unseen-class pixel features; PADing unifies zero-shot semantic, instance, and panoptic segmentation. Zero-shot approaches that replace fixed classifiers with language embeddings can still lack novel-class knowledge and generalize poorly.Source: §2.4 · pp. 4–5
  • Long-tail detection and instance segmentation address class imbalance through resampling, loss reweighting, or decoupled training, but these methods do not currently detect novel classes.Image-level resampling can bias instance co-occurrence; instance- or feature-level resampling is one refinement.Source: §2.4 · p. 5
  • Few-shot detection and segmentation adapt using limited labeled examples, but still require novel-class examples during training, which may be unavailable.Detection examples include two-phase fine-tuning in TFA and separated RPN-feature and RoI-classification training in DeFRCN; segmentation methods use support-image prototypes or support branches.Source: §2.4 · p. 5

3 METHODS: A SURVEY

The survey reviews preliminary detection and segmentation concepts, vision-language models, and related open-vocabulary tasks. It organizes methods across segmentation, video, and 3D by task and shared design patterns, including approaches that use caption data, weak supervision, and cross-modal alignment.

3.1 Preliminary

  • Query-based transformers use object queries and one-to-one Hungarian matching, and are widely used as base models for open-vocabulary tasks despite not directly detecting novel classes.Pixel-based approaches include two-stage detectors with region proposals and one-stage detectors; two-stage methods are more common in open-vocabulary detection. Instance segmentation extends detection with mask heads or clusters semantic predictions.Source: §3.1 · pp. 6–7

3.2 Vision Language Modeling

  • VLM pretraining on large image-text datasets underpins open-vocabulary learning, which transfers VLM knowledge into task models using text embeddings in place of fixed classifiers.Vision embeddings for proposals are compared with base- and novel-class text embeddings; training on base annotations lets the model use VLM semantics alongside learned visual features. CLIP and ALIGN exemplify large-scale image-text pretraining.Source: Figure 5 · p. 6, §3.2 · pp. 7–8

3.3 Open Vocabulary Object Detection

  • Knowledge distillation transfers VLM knowledge into close-set detectors through instance-, global-, relational-, and fine-grained signals, but recognition remains bounded by the teacher VLM,,, [227],.Examples include ViLD’s region-level visual distillation, HierKD’s global alignment, OADP’s object and feature distillation, and caption- or attribute-based supervision.Source: §3.3 · p. 8
  • Region-text pretraining aligns visual and text features to enable novel-class recognition, using caption grounding, pseudo region-text pairs, and large-scale grounding, [232].More text-image pairs can improve novel-class performance, but extra dataset training incurs computation cost.Source: §3.3 · pp. 8–9
  • Training with more balanced data addresses rare and unseen classes through image-level labels, pseudo-boxes, generated examples, and self-training, with OWLv2 reporting gains on LVIS and COCO, [135], [239].Detic uses image-level ImageNet21K supervision, while OWLv2 generates WebLI pseudo-boxes for pretraining before fine-tuning on an open-vocabulary detection dataset.Source: §3.3 · pp. 9–10
  • Prompting adapts VLMs with learned category prompts and region-focused mechanisms, but its performance is limited without additional data,,, [247].Methods address negative proposals, add category descriptions or cached web data, and use region prompting, anchor pre-matching, masked attention, or RoI pruning.Source: §3.3 · p. 10
  • Region-text alignment improves region-feature and text-feature matching during base-class training without extra data, using conditional matching, set matching, or bags of contextual regions,,.Other approaches align co-occurring objects or combine detector scores with VLM predictions,.Source: §3.3 · p. 10

3.3 Open Vocabulary Object Detection; 3.4 Open Vocabulary Segmentation

  • RO-ViT uses cropped positional embeddings to bridge vision-language pre-training and downstream open-vocabulary detection, while segmentation methods align category text with dense visual features.Region-text alignment remains a core research topic; caption-grounding and caption-generation methods also exploit nouns in image captions for segmentation.Source: §3.3 · p. 10, §3.4 · pp. 10–11

3.4 Open Vocabulary Segmentation

  • Segmentation methods generate pseudo labels from dense CLIP features or caption-region matches, while weakly supervised approaches train without pixel-level masks but produce lower-quality results.MaskCLIP+ [120], OVSeg, and XPM use pseudo-labeling; GroupViT, TCL [263], and other methods learn from image-caption data, with mask-free training remaining difficult to use in practice.Source: §3.4 · p. 11
  • Unified segmentation frameworks support semantic, instance, and panoptic tasks, while joint detection-segmentation methods use decoupled decoding at additional computational cost.X-Decoder and FreeSeg address multiple segmentation tasks; OpenSeeD and OpenSD [129] jointly learn from segmentation and detection datasets.Source: §3.4 · p. 11
  • Diffusion methods either use semantically rich intermediate representations for segmentation or generate category prototypes and training images or masks.ODISE trains a decoder on diffusion representations; OVDiff [132] compares inputs with generated prototypes without training, while other methods generate data to augment rare classes.Source: §3.4 · p. 11

3.5 Open Vocabulary Video Understanding

  • Video methods transfer CLIP knowledge through temporal fusion or fine-tuning for classification, and extend open-vocabulary recognition to tracking and video instance segmentation.ActionCLIP [268] adds temporal fusion; ViFi-CLIP shows a simple fine-tuning baseline can be strong. OVTrack, MindVLT, and OpenVIS [272] address tracking or video instance segmentation.Source: §3.5 · p. 12

3.6 Open Vocabulary 3D Understanding

  • 3D recognition methods project point clouds into CLIP-compatible views or align 3D features with image-text representations; 3D detection methods align regional features with text to recognize unseen classes.PointCLIP uses projected depth maps, while ULIP and OpenShape [276] align or scale multimodal data. OV-3DET relies on 2D detector localization; CODA and L3Det [294] outperform it.Source: §3.6 · pp. 12–13
  • 3D scene methods align point-cloud features with language using generated descriptions or back-projected CLIP pixels, while instance pipelines use mask proposals, 2D views, and CLIP for labeling.Open3DIS [302] improves on OpenMask3D across several datasets; CLIP2Scene uses semantic-driven cross-modal contrastive learning and spatial-temporal consistency regularization for driving-scene segmentation.Source: §3.6 · pp. 13–14

3.7 Closely Related Tasks

  • Related tasks include class-agnostic proposals and entity segmentation, open-world detection that incrementally learns novel categories, and open-set panoptic segmentation that labels novel categories as unknown.SAM [42] uses prompting and a large mask dataset for generalized segmentation; open-world detection methods address unknown identification and catastrophic forgetting, while Dual combines known-class discrimination with class-agnostic object prediction.Source: §3.7 · p. 14

4 CHALLENGES AND OUTLOOK

The survey identifies challenges in base-class generalization, training cost, cross-dataset evaluation, and benchmark coverage. It proposes future work spanning video and 3D, task-specific adaptation, incremental learning, and aligning language models with visual systems.

4.1 Challenges

  • Fine-grained feature modeling is needed to reduce base-class overfitting, while more affordable VLM-based training pipelines must balance efficiency against representation capacity.Novel and base classes can differ in shape and semantics, yet detectors may overfit base classes when novel classes are similar. A frozen backbone is one possible cost-saving approach, but may limit representation capacity.Source: §4.1 · p. 14
  • Open-vocabulary learning needs shared cross-dataset models, better metrics for overlapping concepts, and larger datasets to advance beyond benchmark-specific evaluation.Unified models still trail dataset-specific models on some OVD benchmarks, including LVIS; the survey points to SAM-1B [42] as an example of the larger datasets needed.Source: §4.1 · p. 14

4.2 Future Work

  • Future work should build more challenging video benchmarks for open-vocabulary detection and tracking and better align 2D vision-language knowledge with 3D models.Only a few works explore video, and some clips contain few instances; 3D annotation is especially costly for dense prediction, while current solutions focus on projection functions for 2D VLMs.Source: §4.2 · pp. 14–15
  • Task-specific adapters could extend foundation models to custom tasks, with in-context learning explored to connect VLM and LLM knowledge.The survey notes corner cases in medical and aerial imagery despite foundation models’ zero-shot performance on standard datasets, and suggests adapters to use pre-trained knowledge.Source: §4.2 · p. 15
  • Future frameworks should address catastrophic forgetting alongside incremental novel-class detection and align LLM knowledge with visual detectors or segmenters for stronger zero-shot results.Open-world object detection focuses on novel-class localization rather than classification, while LLMs contain broader text concepts than dataset taxonomies, including V3Det [50].Source: §4.2 · p. 15

5 CONCLUSION

The survey presents what appears to be the first detailed review of open-vocabulary learning in computer vision, covering foundational knowledge, more than 50 models, related domains, experiments, challenges, and future directions.

  • The survey covers foundational concepts and introductory knowledge of detection, segmentation, and vision-language pre-training.Source: §5 · p. 15
  • It summarizes more than 50 models across scene-understanding tasks, categorizing methods by technical viewpoint.Source: §5 · p. 15
  • The survey discusses closely related domains and describes experimental settings and fair comparisons of results.Source: §5 · p. 15
  • It identifies challenges and future research directions for open-vocabulary learning.Source: §5 · p. 15

APPENDIX

The appendix compares methods across task settings by describing evaluation datasets, detailing each setting, and tabulating results. For open-vocabulary object detection, it reports datasets, metrics, and method comparisons.

Open Vocabulary Object Detection

  • The appendix compares open-vocabulary detection methods on COCO, LVIS, and V3Det, detailing dataset settings and reporting results in tables.The datasets contain 80 COCO classes, 1,203 LVIS classes, and 13,204 V3Det categories.Source: APPENDIX · p. 15
  • BARON leads with 42.7 APnovel on COCO and 22.6 APr on LVIS using Faster R-CNN with ResNet-50; one-stage methods still significantly trail two-stage detectors.FVLM scores 28.0 APnovel on COCO and 18.6 APr on LVIS using only a frozen backbone and detection data; F-VLM reaches 32.8 APr on LVIS with ResNet50x64, surpassing methods using Swin-B and additional data such as CC3M and ImageNet.Source: APPENDIX · p. 15
  • COCO uses 48 base and 17 novel classes, LVIS uses 866 base and 377 rare novel classes, and V3Det uses 6,709 base and 6,495 novel categories.Evaluation focuses on APnovel alongside APbase, using box mAP at IoU 0.5 for COCO and mask mAP for LVIS.Source: APPENDIX · p. 15

Open Vocabulary Semantic Segmentation

The survey compares open-vocabulary semantic segmentation in same-dataset self-evaluation and cross-dataset evaluation, reporting mIoU and summarizing results across novel-class and cross-dataset benchmarks.

Open Vocabulary Semantic Segmentation

  • Self-evaluation trains and tests on the same dataset, whereas cross-evaluation trains on COCO-Stuff and tests on other datasets.The survey excludes n-fold evaluations from its comparisons because few works adopt that setting.Source: Open Vocabulary Semantic Segmentation · p. 15
  • Benchmarks include ADE20K splits A-847 and A-150, Pascal Context splits PC-459 and PC-59, and Pascal VOC PAS-20.PAS-20b is a harder setting that assigns background only to Pascal VOC pixels having a PC-59 category.Source: Open Vocabulary Semantic Segmentation · p. 16, Table 13 · p. 18
  • Cross-evaluation averages mIoU across all evaluation categories, while self-evaluation reports mIoU on novel classes for fair comparison.In cross-evaluation, all evaluation classes are treated as novel; self-evaluation test sets usually contain both base and novel classes.Source: Open Vocabulary Semantic Segmentation · p. 16
  • Among methods without pixel-level annotations, PACL achieves a much higher score than others and uses extra training data including CC3M, CC12M, and YFCC.The survey notes that X-Decoder uses 4M image-text pairs from Conceptual Captions, SBU Captions, Visual Genome, and COCO Captions.Source: Open Vocabulary Semantic Segmentation · pp. 17–18
  • MaskCLIP+ has the highest novel mIoU on COCO-Stuff, PASCAL-VOC, and PASCAL-Context, while FreeSeg generally leads on base and harmonic mIoU.Harmonic mIoU is the harmonic mean of base-class and novel-class mIoU.Source: Open Vocabulary Semantic Segmentation · p. 16, Table 12 · p. 18
  • SCAN leads A-847 at 14.0 mIoU, SED leads PC-459 at 22.6 and A-150 at 35.2, and ODISE leads PAS-20b at 84.6.X-Decoder records 65.1 on PC-59 and 97.9 on PAS-20.Source: Open Vocabulary Semantic Segmentation · p. 16, Table 13 · p. 18

Open Vocabulary Instance Segmentation

Open-vocabulary instance segmentation is evaluated in constrained and generalized settings, with CGG achieving the best results in both and Mask-free OVIS scoring relatively highly on novel classes without mask labels.

Settings

  • Generalized evaluation combines base and novel categories, requiring the model to classify novel classes and distinguish them from base classes; it is harder than constrained evaluation.Constrained evaluation assesses base and novel categories separately.Source: Open Vocabulary Instance Segmentation · p. 18

Results on COCO

Evaluation Metrics

  • The evaluation uses mask mean Average Precision (mAP), reporting APbase and APnovel separately for constrained evaluation and base, novel, and all-class mAP for generalized evaluation.Source: Open Vocabulary Instance Segmentation · p. 18

Results on COCO

Open Vocabulary Panoptic Segmentation

Models trained on COCO-Panoptic are evaluated zero-shot on ADE20K and COCO using PQ, SQ, and RQ. Results identify different top-PQ methods for ADE20K and COCO seen and unseen classes.

Open Vocabulary Panoptic Segmentation

  • On ADE20K, ODISE-cap achieves 23.4 PQ, 0.8 above the second-best score; on COCO, PADing reaches 41.5 PQ for seen classes and Freeseg reaches 29.8 PQ for unseen classes.The models are trained on COCO-Panoptic and tested on other datasets in a zero-shot manner.Source: Open Vocabulary Panoptic Segmentation · p. 19

Open Vocabulary Video Recognition

Video-recognition methods are commonly evaluated for zero-shot transfer to downstream datasets after Kinetics-400 pretraining. Open-VCLIP leads the reported comparisons, while MAXI performs well without Kinetics-400 annotations.

Open Vocabulary Video Recognition

  • Open-VCLIP performs best on all three downstream datasets, and MAXI performs well without Kinetics-400 annotations.The comparison covers video-classification methods evaluated on downstream datasets such as UCF and HMDB after Kinetics-400 pretraining.Source: Open Vocabulary Video Recognition · p. 19
  • Most existing methods use Kinetics-400 ground truth for training, whereas MAXI does not require those annotations.The reported zero-shot evaluations use downstream datasets such as UCF and HMDB.Source: Open Vocabulary Video Recognition · p. 19

Open Vocabulary Video Instance Segmentation

The section describes an evaluation protocol for open-vocabulary video instance segmentation and reports that MindVLT performs best on both base and novel categories, with stronger improvement on novel categories.

Open Vocabulary 3D Recognition

The section reviews zero-shot classification for open-vocabulary 3D recognition across datasets of different scales and reports that larger training datasets and backbones can improve downstream performance.

Open Vocabulary 3D Recognition

  • Larger-scale OpenShape training data and larger backbones significantly boost performance on downstream 3D recognition datasets.Evaluation covers ModelNet40 (40 common categories), ScanObjectNN (15), and Objaverse-LVIS (1,156 LVIS categories); methods trained on different datasets are tested on all three for fair comparison.Source: Open Vocabulary 3D Recognition · p. 19

Open Vocabulary 3D semantic segmentation

Open-vocabulary 3D semantic segmentation is evaluated on ScanNet and nuScenes using base/novel category splits, with results reported for base, novel, and harmonic mIoU. RegionPLC [297] performs notably on novel categories, indicating generalization to unseen categories.

Open Vocabulary 3D semantic segmentation

  • Evaluation uses ScanNet and nuScenes, with 19 and 15 categories respectively, split into base and novel categories for testing.For example, B15/N4 denotes 15 base categories used for training and four novel categories absent from training.Source: Open Vocabulary 3D semantic segmentation · p. 19
  • RegionPLC [297] achieves remarkable performance on novel categories, indicating a strong ability to generalize to unseen categories.The comparison reports base-category mIoU (mIoU_B), novel-category mIoU (mIoU_N), and harmonic mIoU (hIoU).Source: Open Vocabulary 3D semantic segmentation · p. 19

Related papers

Loading 2306.15880v4…