PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-world Machine Learning: Applications, Challenges, and Opportunities

Jitendra Parmar, Satyendra Singh Chouhan, Vaskar Raychoudhury, Santosh Singh Rathore

arXiv:2105.13448v2cs.LGcs.AI

TL;DR

Closed-world models assume training examples for test classes, so they do not address classes arriving in dynamic settings. This review organizes a decade of OWML work by task, method, and dataset, and identifies research gaps and future directions.

  • Problem

    Closed-world learning assumes each test class is represented in training, limiting its ability to address new classes that arrive in dynamic real-world settings.

    Source: §1 · pp. 1–2

  • Method

    The paper systematically reviews work from the last decade, presenting a task-based OWML classification and examining techniques, datasets, research gaps, and challenges.

    Source: §1 · p. 3, §11 · p. 36

  • Results

    The review classifies OWML tasks in computer vision and image processing and natural language processing, and discusses techniques, datasets, and technology limitations.

    Source: §11 · p. 36

  • Takeaways & Limitations

    The review is intended to help researchers select applicable OWML methodologies and datasets and identify suitable directions for further work.

    Source: §1 · p. 3, §11 · p. 36

    The review reports a lack of end-to-end OWML models and no promising model for unknown-unknown class identification.

    Source: §10 · p. 35

Abstract

from arXiv · show

Traditional machine learning mainly supervised learning, follows the assumptions of closed-world learning, i.e., for each testing class, a training class is available. However, such machine learning models fail to identify the classes which were not available during training time. These classes can be referred to as unseen classes. Whereas open-world machine learning (OWML) deals with unseen classes. In this paper, first, we present an overview of OWML with importance to the real-world context. Next, different dimensions of open-world machine learning are explored and discussed. The area of OWML gained the attention of the research community in the last decade only. We have searched through different online digital libraries and scrutinized the work done in the last decade. This paper presents a systematic review of various techniques for OWML. It also presents the research gaps, challenges, and future directions in open-world machine learning. This paper will help researchers understand the comprehensive developments of OWML and the likelihood of extending the research in suitable areas. It will also help to select applicable methodologies and datasets to explore this further.

1 Introduction

Closed-world models assume testing classes were represented in training, but real-world systems encounter new classes. This paper reviews OWML across CV-IP and NLP, mapping methods and research gaps.

1 Introduction

  • OWML addresses dynamic settings where new classes arrive regularly and closed-world models cannot handle them.The paper cites applications including self-driving cars, healthcare, surveillance, robotics, chatbots, and intelligent assistants.Source: §1 · p. 2
  • The review identifies a gap in broad coverage: earlier reviews were task- or domain-specific, with little coverage of NLP applications.The authors note that existing reviews included malware, person identification, or CV-IP, while OWML was also being used in NLP.Source: §1 · p. 3
  • The paper presents a systematic review and a taxonomy of OWML methods used in computer vision and natural language processing.It also provides tabular summaries of advantages and disadvantages and discusses baseline benchmark algorithms.Source: §1 · p. 3
  • The review analyzes techniques, datasets, research gaps, and challenges to identify future research directions.The stated contributions also include associated fields of OWML and their open-world problems.Source: §1 · p. 3

2 Background and Formal Definition

OWML differs from closed-world learning by recognizing or rejecting unseen classes, and its process can identify rejected classes and update the model. The section also distinguishes it from semi-supervised and transfer learning.

2 Background and Formal Definition

  • OWML is motivated by changing data, and can learn new classes and update an existing model without retraining.The paper distinguishes seen-seen, seen-unseen, and unseen-unseen instances.Source: §2 · p. 4
  • OWML uses seen data for training, tests on seen and unseen data, and is the only paradigm described as able to reject unseen instances.The paper contrasts this with other machine-learning paradigms by their training and testing data and knowledge accumulation or retention.Source: §2 · p. 5
  • Unlike semi-supervised and transfer learning, OWML rejects unseen test instances rather than assuming they belong to known classes.Semi-supervised learning treats unlabelled data as belonging to seen classes; transfer learning transfers knowledge but assumes closed-world testing.Source: §2 · p. 5
  • The formal objective is to classify a test example into one of the seen classes or identify it as an unseen class.The training data consists of labeled instances from the seen classes.Source: §2 · p. 5
  • At time t, a classifier recognizes seen classes or rejects instances into a set that may contain multiple unknown classes.The process then identifies hidden classes in the rejection set, prepares training data, and updates the model with the newly identified data.Source: §2 · pp. 5–6

3 Review Methodology

The review follows a methodical review process, collecting and selecting relevant OWML studies before organizing them by domain and task. It summarizes the reviewed methods and datasets in a task-based taxonomy.

3 Review Methodology

  • The review follows conventional review processes, including a timeline, survey, outcome discussion, challenges, conclusions, and future directions.Its stated review plan also includes examining prior claims, classifying research gaps, and reporting future directions.Source: §3 · p. 6, §3.1 · p. 6

3.1 Review Plan

  • The review plan frames an investigation query, finds relevant tasks and methods, synthesizes studies, and reports results with analytical and tabular comparisons.The authors list eight reporting steps, from exploring initial investigations through writing the survey.Source: §3.1 · p. 6

3.1.1 Investigation Queries

  • The investigation queries cover OWML’s importance, growth, classifications, related domains, tasks, methods, datasets, challenges, and future directions.The authors also collected pertinent information from prominent electronic sources to build the review.Source: §3.1.1 · pp. 6–7

3.1.2 Sources of information and Selection Criteria

  • In approximately all searches, the review retained relevant articles with “openworld” in the title or abstract, following recent articles and technical reports and relying only on notable, reputed sources.Selected articles were categorized into CV-IP and NLP and further classified by task. Most research occurred between 2016 and 2021, and the timeline shows less open-world machine-learning research in NLP than in CV-IP.Source: §3.1.2 · pp. 7–8

3.2 Taxonomy of Open-world Machine Learning

  • The taxonomy organizes CV-IP work into OSR, MCOSC, DUI, and DNC, and NLP work into IL, DUI, KG & KB, and DNC.It catalogs domains, methods, and datasets; the review describes methods used across these task categories.Source: §3.2 · pp. 9–10

4 Open-world Machine Learning in Computer Vision and Image Processing

This section reviews open-world learning for computer-vision and image-processing tasks, covering approaches to reject unknown classes and discover and incorporate novel classes in dynamic environments. It summarizes methods, evaluation settings, software implementations, and challenges, including limitations in risk optimization, training samples, localization, and open-space risk minimization.

4.1 Open-set Recognition (OSR)

  • NNO outperformed SVM, 1-vs-set, and NCM on 50- and 200-category evaluations, while CAP and W-SVM reduce open-space risk and incremental methods extend recognition.NNO rejects objects when all class scores are zero, then treats collected novel objects as new classes; ONNO, ONCM, and ONBC also outperformed existing methods in the reported experiments.Source: §4.1 · pp. 11–12

4.2 Multi-class Open-set Classification (MCOSC)

  • PI-SVM improved F-Measure by 12 to 22 percent, while MLOSR [100] performed better than existing networks for open-set recognition.PI-SVM was evaluated in open-set MNIST and cross-dataset settings; MLOSR combines classification, reconstruction, and EVT-based modeling of reconstruction-error tails.Source: §4.2 · pp. 13–14

4.3 Discovery of Unseen Instances (DUI)

  • OpenMax [7] estimates unknown-class likelihood and rejects unrelated images; CROSR [156] outperformed several deep classifiers, while adversarial training improved end-to-end strength but weakened benign out-of-distribution detection.NGC [147] filters out-of-distribution examples without additional training and improved over existing methods; CROSR combines known-class classification with unseen-class detection through reconstruction.Source: §4.3 · pp. 15–16

4.4 Detection of Novel Classes (DNC)

  • A Bayesian Gaussian-embedding few-shot approach [144] discovers unknown classes and adapts to new classes with limited labels, improving H-measure by 12 percent.SIM is a semi-supervised stream classifier for high-dimensional data that classifies and detects novel classes in image and text streams.Source: §4.4 · pp. 16–17

4.5 Available Software Packages and Implementations

  • The reviewed software includes methods for unknown-class rejection, open-space risk control, open-set recognition, and discovering new classes.Examples named include 1-vs-rest, NNO, C2AE, MDCC, PI-SVM, and W-SVM.Source: §4.5 · p. 17

4.6 Discussion

  • Real-world deployment still needs a generic framework for dynamic, multi-domain inputs, with localization, open-space-risk minimization, and prior knowledge supporting continual learning.The discussion identifies unpredictable input size and frequency as a challenge and points to interactive, end-to-end systems for recognizing multiple objects in open space.Source: §4.6 · p. 17
  • Using past knowledge to recognize unseen objects in dynamic environments can improve accuracy and realism, while the knowledge base must be updated incrementally.Systems must also handle empirical and open-space risks to achieve realistic results.Source: §4.6 · p. 18
  • Open-space and empirical risk parameters are not optimized, limiting models’ ability to adapt generalization or specialization.Source: §4.6 · p. 19
  • Limited training sampling and the lack of localization in most recommended methods make real-world impact uncertain and hinder handling images with multiple objects.Source: §4.6 · p. 19
  • The authors note the absence of a mechanism to minimize open-space risk and suggest dictionary learning-based algorithms for open-set recognition.Source: §4.6 · p. 19

5 Open-world Machine Learning in Natural Language Processing (NLP)

The section surveys open-world NLP tasks, methods, datasets, and software, then identifies challenges in recognizing unseen text and adapting knowledge in dynamic settings.

5 Open-world Machine Learning in Natural Language Processing (NLP)

  • NLP open-world tasks span text classification and analysis, including sentiment, spam filtering, movie-genre reviews, and document classification.The task summary table identifies methodologies, datasets, and proposed results for these tasks.Source: §5 · p. 19, Table 6 · p. 19

5.1 Incremental Learning (IL)

  • Incremental text learning retains earlier knowledge as new tasks arrive; CBS-SVM detects unknown classes, adds them cumulatively, and outperforms listed baselines across openness levels.CBS-SVM builds a classifier for a detected new class and updates the existing classifier without retraining the whole system.Source: §5.1 · p. 20

5.2 Discovery of Unseen Instances (DUI)

  • Unseen-instance discovery methods include DOC’s CNN with a 1-vs-rest sigmoid layer and Gaussian fitting, alongside intent detectors and classifiers for novel classes.The section also surveys approaches such as TOP-ID, SMDN, and ADVIN for discovering unseen intents or classes.Source: §5.2 · pp. 20, 22, §5.4 · p. 24
  • IDCF improves results for few-shot and unseen users over several inductive models, while achieving RMSE close to transductive collaborative-filtering models.The reported evaluations use explicit-feedback MovieLens-100K, MovieLens-1M, and Douban, and implicit-feedback Amazon-Beauty and AmazonBooks.Source: §5.2 · p. 23

5.3 Knowledge-graph & Knowledge-base (KG & KB)

  • Open-world knowledge-graph completion addresses missing or unknown entities; ConMask masks relation-relevant text, while LILI learns and adds knowledge through chatbot conversations.Caps-OWKG uses triplets to represent relationships and reports better outcomes than the compared techniques on FB15k-237 and DBPedia50k.Source: §5.3 · pp. 23–24

5.4 Detection of Novel Classes (DNC)

  • ADVIN discovers novel domains and intents from unlabeled text in three stages and reports significantly better overall performance on four benchmark datasets.Its stages are discovery, knowledge transfer, and linking intents to domains.Source: §5.4 · p. 24

5.5 Available Software Packages and Implementations

  • The section lists NLP software packages for open-set recognition, unseen-class identification, unseen-entity detection, and word embeddings.It identifies OSDN, DOC, ConMask, Word2Vec, and GloVe as examples.Source: §5.5 · p. 25, Table 8 · p. 25

5.6 Discussion

  • NLP open-world systems face semantic-similarity challenges, reduced accuracy as openness increases, and limited large-scale evaluation for real-time use.The discussion also notes that many methods struggle to distinguish unseen intents from seen intents with similar meanings.Source: §5.6 · pp. 25–26

6 Datasets Used in Open-world Machine Learning

The section catalogs datasets used in open-world machine learning across computer vision and NLP, describing their properties and showing their classification by domain and year.

6 Datasets Used in Open-world Machine Learning

  • The reviewed datasets span computer vision and image processing and NLP, with a figure classifying them by domain and proposed year.The section introduces both commonly used and publicly available benchmarks.Source: §6 · p. 26, Figure 6 · p. 27
  • Vision benchmarks described include Caltech-256, MNIST, Fashion-MNIST, ImageNet, Tiny-ImageNet, and CIFAR-10.Their descriptions cover image counts, categories, resolutions, and dataset organization.Source: §6 · pp. 26–27
  • Text datasets include 20-NewsGroup and Amazon Product Reviews, alongside 50-Class Reviews for product-domain review classification.The passages describe the size or category structure of these review and newsgroup collections.Source: §6 · p. 28
  • WordNet provides lexical semantic relationships, while SwDA supplies telephone-dialogue turns with semantic, syntactic, and pragmatic tags.The section also points readers to repositories for publicly available datasets.Source: §6 · p. 28

7 Baseline Algorithms Used in Open-world Machine Learning

The section explains baseline methods used in open-world learning, focusing on incremental class handling, class-centroid classification, and rejection of unknown classes.

7.1 Center-Based Similarity (CBS) [32]

  • CBS incrementally extends 1-vs-rest classifiers to classify existing classes and recognize unseen ones as new-class data arrives.Its workflow searches for comparable classes, isolates the new class, builds a classifier, and updates relevant existing classifiers.Source: §7.1 · p. 29

7.2 Incremental Class Learning [32]

  • Incremental class learning uses a similarity threshold to find prior classes resembling a new class before building its novel classifier.The method counts positive classifications of new-class instances by existing classifiers and compares them with the threshold.Source: §7.2 · p. 30

7.3 Nearest Class Mean (NCM) [84, 111]

  • NCM represents each class by the mean of its feature vectors and classifies an image by its closest class centroid.The section describes computing one centroid for each class from its labeled images.Source: §7.3 · p. 30

7.4 1-vs-rest [110]

  • Classical multiclass outputs cannot reject unknown classes, so 1-vs-rest normalizes per-class sigmoid outputs and rejects inputs below a class threshold.Each sigmoid treats one class as positive and all remaining classes as negative.Source: §7.4 · pp. 30–31

7 Baseline Algorithms Used in Open-world Machine Learning

  • The surveyed baselines recur across computer vision and NLP, with extensions such as NNO addressing scaling burdens associated with NCM.The section also notes use of 1-vs-rest for unknown-data identification and as part of other frameworks.Source: §7.4 · p. 31

8 Related Areas

The section relates open-world learning to transfer, active, lifelong, and multi-task learning, emphasizing knowledge reuse across tasks and the challenges of retaining and transferring it efficiently.

8.1 Transfer Learning

  • Transfer learning uses knowledge from a reference task to support a new task, often across comparable but non-identical tasks, fields, or distributions.The section distinguishes this transfer of information from learning confined to a single task setting.Source: §8.1 · pp. 31–32

8.2 Active Learning

  • Active learning lets an algorithm choose learning data or strategy and can provide high accuracy with a small amount of labeled data.The section names membership-query synthesis, stream-based selective sampling, and pool-based active learning as scenarios.Source: §8.2 · p. 32

8.3 Lifelong Learning/Continual Machine Learning

  • Lifelong learning retains selected knowledge from different sources and transfers it to improve learning on future tasks.The section discusses supervised, reinforcement, and unsupervised variants and gives examples including EBNN and NELL.Source: §8.3 · pp. 32–33

8.3.1 Challenges and Benefits of Lifelong Learning Models [126].

  • Lifelong systems must retain valid, non-erroneous knowledge within finite memory and computational limits while transferring it efficiently across heterogeneous domains.The section also identifies input diversity, scalability, and computational cost as challenges.Source: §8.3.1 · pp. 33–34

8.4 Multi-task Learning

  • Multi-task learning jointly acquires associated tasks to build a more reliable representation from shared knowledge and reduce overfitting on individual tasks.The section describes classification and ranking as two categories of deep-learning multi-task approaches.Source: §8.4 · p. 34

9 Research Challenges

Open-world learning faces challenges from growing, unstructured data and the arrival of unseen categories, as well as from building systems that can learn and update dynamically. The paper identifies gaps in novel-class discovery, open-space risk management, and retention of previously acquired knowledge.

Incremental Volume of Data

  • Growing, unstructured data and continuous arrivals of unseen categories make it difficult to manage data and identify classes across domains.The paper notes a lack of mechanisms for handling real-world data.Source: §9 · pp. 34–35

Identifying a Novel Classes

  • After rejecting unseen instances, a system must group them into classes to acquire novel classes.The framework must find instances belonging to each class.Source: §9 · p. 35

Updating a Knowledge-Base; Open-space Risk

  • Runtime knowledge-base updates must add new domains and classes for later predictions without retraining the entire model, while also addressing open-space risk.The paper describes open-space as space away from positive training examples and calls for reducing its risk as openness increases.Source: §9 · p. 35

Open Framework

  • An end-to-end open-world framework must dynamically coordinate modules for discovering unseen instances and identifying novel classes while addressing seen and unseen classes.The paper notes that synchronizing modules using different methods is complicated.Source: §9 · p. 35

Efficient Discovery of Novel Classes

  • Existing models can discover unseen instances, but few identify novel classes or efficiently determine how many unseen classes there are.Source: §9 · p. 35

Retention of Obtained Knowledge

  • Few models continuously update knowledge while learning new tasks and use previously acquired knowledge for subsequent predictions.Source: §9 · p. 35

10 Future Directions

The paper points to open-world models, unknown-class rejection, and identification of rejected classes as areas for future development. It frames improvement around model capability, rejection, and identification of new classes.

Future Directions

  • The review identifies model capability, rejection, and new-class identification as three aspects for improving open-world machine learning.Source: §10 · p. 35

Open-world Models

  • Existing open-world models address problems in parts, and the paper reports no promising model for identifying unknown-unknown classes.End-to-end models could strengthen classification of known-known and known-unknown classes; existing unknown-unknown methods are worth extending.Source: §10 · p. 35

Rejection of Unknown Classes; Identification of Unseen Classes

  • Few works reject unknown-unknown classes, and the paper calls for improved rejection and mechanisms to identify classes in rejected data.It also calls for models that identify the number of hidden classes in rejected data.Source: §10 · p. 36

11 Conclusion

The paper reviews open-world machine-learning work from the last decade and classifies tasks in computer vision and image processing, and natural language processing. It discusses techniques, datasets, limitations, and future extensions.

Conclusion

  • The review examines open-world machine-learning work from the last decade and discusses its prominence and real-life applications.Source: §11 · p. 36
  • It provides a task-based classification for computer vision and image processing, and natural language processing, and discusses techniques and datasets used in open-world machine learning.Source: §11 · p. 36
  • The review analyzes technology limitations to support promising future extensions and aims to help readers understand open-world settings, their operation, and associated challenges.Source: §11 · p. 36
Loading 2105.13448v2…