PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Open Vocabulary Learning: A Survey

Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, Dacheng Tao

arXiv:2306.15880v4cs.CVcs.AI

TL;DR

Open vocabulary learning addresses the limitations of close-set visual scene understanding, where predefined labels, incomplete annotations, and costly labeling restrict recognition of real-world categories. This survey synthesizes methods, settings, benchmarks, and future directions across open vocabulary detection, segmentation, and related scene-understanding tasks. It reviews more than 50 models and compares results across tasks and datasets.

  • Problem

    Close-set visual models recognize only predefined training categories, while real scenes contain more concepts and require costly, incomplete annotations.

  • Method

    The paper surveys open vocabulary learning by reviewing related concepts, task settings, methods, design elements, benchmarks, and future research directions.

  • Results

    The survey examines more than 50 models and compares recent detection and segmentation approaches across commonly used datasets and benchmarks.

  • Takeaways & Limitations

    Open vocabulary learning is presented as a broad research area spanning detection, segmentation, tracking, 3D understanding, and vision-language-based approaches.

  • Takeaways & Limitations

    Open vocabulary video detection and tracking remain underexplored, with few works evaluated on simple scenes; more dynamic and challenging video datasets are needed.

Abstract

from arXiv · show

In the field of visual scene understanding, deep neural networks have made impressive advancements in various core tasks like segmentation, tracking, and detection. However, most approaches operate on the close-set assumption, meaning that the model can only identify pre-defined categories that are present in the training set. Recently, open vocabulary settings were proposed due to the rapid progress of vision language pre-training. These new approaches seek to locate and recognize categories beyond the annotated label space. The open vocabulary approach is more general, practical, and effective compared to weakly supervised and zero-shot settings. This paper provides a thorough review of open vocabulary learning, summarizing and analyzing recent developments in the field. In particular, we begin by comparing it to related concepts such as zero-shot learning, open-set recognition, and out-of-distribution detection. Then, we review several closely related tasks in the case of segmentation and detection, including long-tail problems, few-shot, and zero-shot settings. For the method survey, we first present the basic knowledge of detection and segmentation in close-set as the preliminary knowledge. Next, we examine various scenarios in which open vocabulary learning is used, identifying common design elements and core ideas. Then, we compare the recent detection and segmentation approaches in commonly used datasets and benchmarks. Finally, we conclude with insights, issues, and discussions regarding future research directions. To our knowledge, this is the first comprehensive literature review of open vocabulary learning. We keep tracing related works at https://github.com/jianzongwu/Awesome-Open-Vocabulary.

1 INTRODUCTION

Open vocabulary learning addresses the limits of close-set vision models by using language and vision-language knowledge to recognize novel categories. This survey organizes the field’s tasks, methods, benchmarks, and future directions.

  • Motivation: Close-set models struggle with real-world scenes because annotations cover limited categories and extending them requires costly labeling.COCO, for example, contains 80 categories, while natural images may contain many more object types.
  • Related Settings: Zero-shot learning generalizes from seen to unseen classes without unseen-class annotations, often constructing classifiers through word-embedding projection.Unseen objects are typically absent from training examples and treated as background.
  • Open Vocabulary Learning: Open vocabulary learning extends models from base to novel classes while allowing visual-related language data, such as captions, as auxiliary supervision.Language data is cheaper to obtain than box or mask annotations and can contain novel class names and attributes.
  • Vision-Language Knowledge: Vision-language models align images and language in a shared feature space, enabling open vocabulary methods to reduce the distinction between close-set and open-set recognition.The survey notes that CLIP is an important example of this pre-trained alignment.
  • Survey Scope: The survey systematically reviews open vocabulary detection, segmentation, video understanding, and 3D scene understanding, covering methods, benchmarks, challenges, and future directions.It also compares open vocabulary learning with related concepts and tracks representative works over time.

2 BACKGROUND

Open vocabulary learning recognizes novel classes using expanded language knowledge alongside visual data, distinguishing it from settings that only reject or classify unseen inputs. The survey traces its history, formalizes its data setting, and reviews applications and related paradigms.

  • Formal Setting: Open vocabulary training combines standard data-label pairs with vision-aware language vocabulary, while evaluation labels include both base and novel classes.During evaluation, predictions must cover labels drawn from either the base set or the novel set.
  • Concept Definitions: Open-set learning identifies known classes and rejects unseen classes as a single unknown category, without classifying those unknowns further.Open-world learning additionally supports incremental recognition and learning of new categories over time.
  • Concept Definitions: Zero-shot learning requires models to classify unseen classes into specific categories, whereas open vocabulary learning uses broader language knowledge to support novel-class recognition.Open vocabulary methods may use low-cost image-text data or pre-trained vision-language models such as CLIP.
  • History and Roadmap: The field developed from joint image-pixel and word-concept embeddings through caption-based detection and CLIP-based detection and segmentation.The timeline also reflects growing use of foundation models, vision-language models, and more recently diffusion models.
  • Tasks and Related Domains: Open vocabulary learning spans detection, segmentation, video understanding, and 3D scene understanding, with task-specific metrics such as mAP, mIoU, and PQ.Related research includes open-set recognition, out-of-distribution detection, zero-shot learning, and few-shot learning.

3 METHODS: A SURVEY

This section surveys open vocabulary methods across detection, segmentation, video, and 3D understanding, after reviewing close-set and vision-language foundations. It organizes approaches around how they transfer or align visual and language knowledge to recognize novel classes.

  • Survey scope: The survey covers six subsidiary tasks, including object detection, segmentation, video understanding, 3D scene understanding, and closely related tasks.It records and compares representative works while listing numerous additional works in tables.
  • Common architecture: A common open vocabulary architecture compares predicted box or mask embeddings with VLM-generated class embeddings and selects the highest dot-product class.Training exposes the system to only a limited set of base classes, while classification can use a broader human-defined class set.
  • Preliminaries: Close-set detection and segmentation foundations include semantic pixel classification, two-stage or one-stage detection, and top-down or bottom-up instance segmentation.Transformer-based query architectures provide a flexible alternative, using object queries and bipartite matching for assignments during training.
  • Method taxonomy: The survey reviews five open vocabulary method categories: knowledge distillation, region text pre-training, balanced-data training, prompting modeling, and region text alignment.These categories summarize recurring strategies for extending close-set detectors and segmenters with vision-language knowledge.
  • Knowledge transfer and alignment: Knowledge distillation transfers VLM knowledge into close-set detectors, while region text pre-training learns alignment from large-scale image-text pairs for novel-class recognition.ViLD uses text and image branches for instance-level visual-to-visual distillation, whereas OVR-CNN uses caption data with grounding, masked language modeling, and image-text matching.
  • Prompting and multi-task learning: Prompt modeling adapts VLM text embeddings through learned prompts, while multi-task approaches jointly use detection and segmentation data but incur extra computation.Examples include DetPro, PromptDet, CORA, OpenSeeD, and OpenSD.

4 CHALLENGES AND OUTLOOK

The survey identifies unresolved challenges in open vocabulary scene understanding, including base-class overfitting, high training costs, limited temporal evaluation, and incomplete 3D and custom-task adaptation.

  • Base Classes Over-fitting Issues: Open vocabulary models can overfit base classes when novel classes have similar shapes or semantics, motivating finer-grained feature modeling.Parts and attributes are identified as possible sources of more discriminative features.
  • Training Costs: State-of-the-art methods often require expensive, large-scale pre-training data, while frozen backbones reduce costs but may limit representation capacity.
  • Explore Temporal Information: Open vocabulary video detection and tracking remain underexplored, with existing clips often too simple to test models in dynamic scenarios.The survey calls for more challenging video datasets to evaluate vision-language models more fully.
  • 3D Open Vocabulary Scene Understanding: 3D open vocabulary scene understanding is especially urgent because point-cloud annotation is expensive, and current methods mainly project 2D vision-language knowledge into 3D.Future work should develop improved alignment of 2D model knowledge with 3D models.
  • Explore Foundation Models With Specific Adapter For Custom Tasks: Custom domains such as medical and aerial imagery require task-specific adapters to address corner cases while leveraging pre-trained foundation-model knowledge.The survey also highlights in-context learning as a possible way to connect vision-language and language-model knowledge.
  • Combining with Incremental Learning: Incremental open-world data creates a need to combine open vocabulary learning with incremental learning, while avoiding catastrophic forgetting.

5 CONCLUSION

The survey reviews open vocabulary learning in computer vision as a broad synthesis of methods, evaluations, related domains, challenges, and future directions.

  • The survey examines recent open vocabulary learning developments and presents background on detection, segmentation, and vision-language pre-training.
  • It categorizes more than 50 models across scene-understanding tasks according to their technical viewpoints.
  • The survey compares experimental settings and results in detailed tables, then summarizes challenges and future research directions.

APPENDIX

The appendix systematically organizes task settings, datasets, methods, and comparative results for open vocabulary learning.

  • The appendix introduces datasets for each setting before presenting setting-specific details and detailed result tables.
  • Open vocabulary object detection commonly evaluates OVR-CNN and ViLD on COCO, LVIS, and V3Det.
  • COCO uses 48 base and 17 novel classes, while LVIS uses 866 frequent or common base categories and 377 rare novel categories.
  • COCO open vocabulary detection uses base-class annotations for training and evaluates both base and novel categories at test time.

Open Vocabulary Semantic Segmentation

Open vocabulary semantic segmentation is evaluated in self-evaluation and cross-evaluation settings using mIoU, with reported results varying across datasets and supervision regimes.

  • Settings: Self-evaluation trains and tests within one dataset, using base annotations for training and evaluating on base and novel categories.ZegFormer and MaskCLIP+ are examples of methods using this setting.
  • Settings: Cross-evaluation tests methods across datasets including ADE20K, Pascal Context, and Pascal VOC.
  • Datasets: ADE20K provides A-847 and A-150 evaluation splits, while Pascal Context provides PC-459 and PC-59 category settings.
  • Evaluation Metrics: mIoU is the evaluation metric; cross-evaluation averages IoU over all evaluation categories, whereas self-evaluation reports novel-class mIoU for fair comparison.
  • Results under the Self-evaluation Setting: MaskCLIP+ achieves the highest novel mIoU across COCO-Stuff, PASCAL-VOC, and PASCAL-Context in self-evaluation, while FreeSeg generally leads base and harmonic mIoU.
  • Results under the Cross-evaluation Setting: In cross-evaluation, SCAN reaches 14.0 mIoU on A-847, SED reaches 22.6 on PC-459 and 35.2 on A-150, and ODISE reaches 84.6 on PAS-20b.
  • Results under the Cross-evaluation Setting: X-Decoder achieves 65.1 mIoU on PC-59 and 97.9 on PAS-20, while some methods use millions of extra image-text pairs for training.X-Decoder uses 4M image-text pairs; PACL uses CC3M, CC12M, and YFCC without pixel-level annotations.

Open Vocabulary Instance Segmentation

Open vocabulary instance segmentation evaluates base and novel categories under constrained and generalized settings. On COCO, CGG performs best in both settings without pretrained VLMs or extra data, while Mask-free OVIS performs strongly on novel classes without mask labels.

  • Evaluation Settings: The constrained setting evaluates base and novel categories separately, whereas the generalized setting evaluates them together and requires distinguishing novel from base classes.The generalized setting is harder because all category names are provided simultaneously.
  • Evaluation Metrics: Mask mean Average Precision (mAP) is used, with APbase and APnovel reported separately for constrained evaluation and base, novel, and all-class mAP reported for generalized evaluation.
  • COCO Results: CGG achieves the best COCO results in both constrained and generalized open vocabulary instance segmentation settings without pretrained VLMs or extra data.
  • COCO Results: Mask-free OVIS obtains a relatively high novel-class score without using mask labels.

Open Vocabulary Panoptic Segmentation

Open vocabulary panoptic segmentation is evaluated across ADE20K and COCO using PQ, SQ, and RQ. Different methods lead on different datasets and class groups, with ODISE-cap best on ADE20K overall, PADing best for seen COCO classes, and Freeseg best for unseen COCO classes.

  • ADE20K Results: ODISE-cap achieves the best ADE20K Panoptic Quality (PQ) score of 23.4, exceeding the second-best score by 0.8.
  • COCO Results: On COCO, PADing achieves a PQ of 41.5 for seen classes, while Freeseg achieves the highest unseen-class PQ of 29.8.
  • Evaluation: The evaluation reports Panoptic Quality (PQ), Segmentation Quality (SQ), and Recognition Quality (RQ) for models trained on COCO-Panoptic and tested on ADE20K and COCO.

Open Vocabulary Video Recognition

Open vocabulary video recognition commonly tests zero-shot transfer from Kinetics-400 to downstream datasets. Open-VCLIP performs best across all three downstream datasets, while MAXI performs well without Kinetics-400 ground-truth annotations.

  • Results Comparison: Open-VCLIP performs best on all three downstream datasets in the reported comparison.
  • Training Setting: MAXI performs well even without using Kinetics-400 ground-truth annotations during training.
  • Evaluation Setting: Existing methods typically evaluate zero-shot capability on downstream datasets such as UCF and HMDB after pretraining on Kinetics-400.

Open Vocabulary Video Instance Segmentation

Open vocabulary video instance segmentation is evaluated on LV-VIS, whose 1,212 categories are divided into 659 base and 553 novel categories. MindVLT achieves the best results for both groups and shows stronger improvement on novel categories.

  • Evaluation Setting: LV-VIS contains 659 base categories and 553 novel categories inherited from LVIS for evaluating open vocabulary video instance segmentation.The dataset is used only for evaluation, not training.
  • Evaluation Metrics: Mean Average Precision (mAP) is reported separately as mAPbase and mAPnovel for base and novel categories.
  • Results Comparison: MindVLT achieves the best results on both base and novel categories in the reported comparison.
  • Results Comparison: MindVLT shows stronger improvement on the novel categories.

Open Vocabulary 3D Recognition

Open vocabulary 3D recognition is evaluated across three datasets with different category scales, and larger pretraining data and backbones improve downstream performance.

  • Evaluation spans ModelNet40, ScanObjectNN, and Objaverse-LVIS, containing 40, 15, and 1,156 categories, respectively.
  • Methods are trained on different datasets but tested across all three datasets for fair comparison.
  • Larger OpenShape datasets and larger backbones significantly boost performance on downstream testing datasets.

Open Vocabulary 3D semantic segmentation

Open vocabulary 3D semantic segmentation is evaluated on ScanNet and nuScenes using base–novel category splits. RegionPLC achieves notable performance on novel categories, indicating generalization to unseen classes.

  • Settings: ScanNet and nuScenes cover broad application scenarios with 19 and 15 categories, respectively.
  • Settings: Categories are split into base and novel groups, such as B15/N4, where four novel categories are absent during training.
  • Results Comparison: Performance is compared using base-category mIoU, novel-category mIoU, and harmonic mIoU.
  • Results Comparison: RegionPLC achieves remarkable performance on novel categories, indicating strong generalization to unseen categories.
Loading 2306.15880v4…