Source-linked AI summary
Towards Open World Recognition
Abhijit Bendale, Terrance Boult
TL;DR
Open world recognition addresses visual recognition in dynamic environments where unknown categories must be detected and learned incrementally without prolonged retraining. The paper formalizes the problem, proves risk-control properties for thresholded distance-based functions, and proposes NNO with an evaluation protocol. NNO consistently outperforms NCM on open world tasks while remaining comparable on closed-set recognition.
Problem
Recognition systems commonly assume a static closed world, despite needing to detect novel categories, update classifiers, and operate with unseen classes in real environments.
Method
The paper formalizes open world recognition, extends NCM into NNO, proves threshold-based open space risk control, and introduces an evaluation protocol.
Results
NNO consistently outperforms NCM on open world recognition tasks, achieves almost 74% improvement over OS-NCM on the largest reported task, and remains comparable on closed-set recognition.
Takeaways & Limitations
NNO supports scalable systems that incrementally add classes and remain robust to unseen categories in dynamic environments where minimum downtime is desired.
Takeaways & Limitations
The theory's lower-dimensional risk guarantee may correspond to exponentially increasing open space risk in the original space as missing dimensions increase.
Abstract
from arXiv · showhide
With the of advent rich classification models and high computational power visual recognition systems have found many operational applications. Recognition in the real world poses multiple challenges that are not apparent in controlled lab environments. The datasets are dynamic and novel categories must be continuously detected and then added. At prediction time, a trained system has to deal with myriad unseen categories. Operational systems require minimum down time, even to learn. To handle these operational issues, we present the problem of Open World recognition and formally define it. We prove that thresholding sums of monotonically decreasing functions of distances in linearly transformed feature space can balance "open space risk" and empirical risk. Our theory extends existing algorithms for open world recognition. We present a protocol for evaluation of open world recognition systems. We present the Nearest Non-Outlier (NNO) algorithm which evolves model efficiently, adding object categories incrementally while detecting outliers and managing open space risk. We perform experiments on the ImageNet dataset with 1.2M+ images to validate the effectiveness of our method on large scale visual recognition tasks. NNO consistently yields superior results on open world recognition.
1 Introduction
Open world recognition addresses the mismatch between dynamic real-world environments and recognition systems that assume all categories are known in advance. The paper formalizes this setting, develops risk-controlled recognition and incremental learning, and evaluates NNO for scalable adaptation.
- Motivation: Real-world recognition systems must continuously update with new categories while handling unseen categories during prediction.Operational systems also face constraints on downtime during learning.
- Problem: Open world recognition extends open set recognition by requiring continuous novelty detection and incremental incorporation of newly labeled classes.The system must progress from unknown detection to labeling and classifier extension.
- Problem: Existing closed-world classifiers are poorly matched to open space because their probability normalization assumes that all classes are known.The paper specifically identifies Softmax-based NCM probabilities as failing to account for open space.
- Contributions: The paper formally defines open world recognition and extends NCM into NNO to balance open space risk with recognition accuracy while incrementally adding categories.NNO is designed to detect outliers and extend the multiclass classifier without requiring complete retraining.
- Contributions: The paper also proves a risk-control result for thresholded, monotonically decreasing distance functions and introduces an evaluation protocol for open world recognition.Experiments use ImageNet to assess the proposed approach on large-scale recognition tasks.
2 Related Work
Related work spans incremental learning, scalable classification, open set recognition, and nearest class mean methods. The paper positions its approach as combining these capabilities while avoiding repeated full retraining.
- Incremental Learning: Incremental SVM methods support adding instances or classes but can incur expensive update processes that limit large-scale use.The related work discusses binary and multiclass incremental SVM extensions and their computational drawbacks.
- Scalable Learning: Scalable classification methods such as label trees and deep models target many categories but generally assume the complete training data is available beforehand.This assumption makes adaptation to dynamic learning scenarios difficult.
- Open Set Learning: Open set recognition handles unknown classes at test time by balancing open space risk against empirical error, but some calibration approaches do not scale.The cited work includes compact abating probability models and Weibull-based calibration.
- Positioning: Figure 2 organizes prior work across open set learning, incremental learning, and scalability, the three capabilities targeted jointly by this paper.The proposed system is described as learning categories incrementally without retraining whenever a new category arrives.
- Nearest Class Mean: NCM classifiers associate transformed image features with class means and provide a distance-based foundation for scalable visual recognition.The paper reviews Euclidean and low-rank Mahalanobis distances and the probabilistic NCM formulation.
3 Open World Recognition
The paper defines open world recognition as a system that recognizes known classes, detects unknowns, labels novel data, and incrementally adds new classes. Its objective combines empirical performance with control of open space risk.
- Risk Objective: Open space risk measures the relative amount of positively labeled open space, and the objective minimizes it together with empirical risk.Open space is defined as regions sufficiently far from known positive training samples.
- Definition: Open world recognition models known classes with open set recognition functions and uses a novelty detector to assign unknown inputs the zero label.The formal definition represents the system as a tuple containing recognition, novelty detection, labeling, and incremental learning components.
- Definition: Novel unknown data are labeled into new classes, after which the known-class set expands from K_t to K_t+1.The labeling process is assumed to produce supervised labels for newly detected data.
- Definition: Incremental learning adds new per-class recognition functions to the existing function vector rather than rebuilding the entire classifier representation.The formal function I_t maps the current recognition functions and labeled data to an expanded set.
- Novelty Detection: A minimum threshold on the highest class score can serve as a novelty detector, rejecting inputs whose best recognition score does not exceed τ.The paper states that this thresholding approach can manage open space risk.
4 Opening existing algorithms
The paper extends open-set risk theory to open-world recognition and uses it to motivate NNO, an incrementally updateable recognition system that rejects unknown inputs. Its guarantees depend on distance-decreasing recognition functions, suitable transformations, and carefully chosen thresholds rather than generic probability rejection.
- Opening existing algorithms: Existing EVT-calibrated 1-vs-rest RBF SVM methods are poorly suited to the incremental updates and scalability required by open-world recognition.The paper therefore pursues an alternative based on non-negative combinations of abating recognition functions.
- Open-space risk theory: The same risk-management result extends to linearly transformed feature spaces, including projections onto lower-dimensional subspaces.The transformed-space theorem transfers bounded open-space risk back to the original space, but the bound can worsen exponentially with omitted dimensions when projected-space risk is nonzero.
- Open-space risk theory: Thresholding a non-negative weighted sum of CAP models can limit open-space risk to any desired level, including zero.The proof bounds the thresholded labeled region through a finite union of compact regions.
- Nearest Non-Outlier (NNO): Softmax normalization is unsuitable for open-world rejection because the nearest class can retain probability near 1 far from all class means, preventing score decay.Adding classes also changes normalization factors, making probabilities difficult to interpret and threshold consistently.
- Nearest Non-Outlier (NNO): NNO adapts NCM with per-class distance-based recognition functions, rejecting inputs when a class score is zero and labeling an input novel when all classes reject it.After human labeling, NNO computes the new class mean and appends it to its vector of class means.
- Nearest Non-Outlier (NNO): NNO satisfies the paper’s open-world definition because its measurable recognition, rejection, human-labeling, and incremental-learning components are consistent and scalable.The construction relies on the preceding theorems for its per-class recognition functions.
5 Experiments
The experiments evaluate open-world recognition incrementally on ImageNet, varying known and unknown categories while comparing NCM-based classifiers. NNO is assessed for scalability, feature robustness, and performance against open-set baselines.
- Dataset and Features: ImageNet 2010 provides 1.2M training images, 50K validation images, and 150K test images across 1K visual categories.Its large category count supports incremental and open-world evaluation.
- Open World Evaluation Protocol: The protocol splits test categories into known and unknown sets, evaluates closed- and open-set performance at each incremental stage, and varies the number of unknown categories.The experiments use 500 known and 500 unknown categories at the largest stage.
- Experimental Setup: The study compares CS-NCM, OS-NCM, and OS-NNO while incrementally adding categories after metric learning.The first experiment starts with 50 categories and adds 50-category increments; the second starts with 200 and adds 100-category increments to reach 500.
- Results: 74% improvement was observed for OS-NNO over OS-NCM on the largest task with 500 training categories and 1,000 testing categories.Repeated folds produced a standard deviation of approximately ±1%.
- Efficiency: Initial metric learning required 15 hours for 50 categories and 22 hours for 200 categories, whereas adding new classes mainly requires fast mean computation after metric learning.The update process is dominated primarily by feature extraction and related processing.
6 Discussion
NNO consistently improves open-world recognition while remaining comparable to NCM in closed-set testing, supporting incremental learning in dynamic environments. Results also suggest that increasing category counts makes closed-world recognition increasingly resemble open-world recognition, while broader feature and application evaluation remains future work.
- NNO consistently outperforms NCM on open-world tasks while remaining comparable on closed-set recognition.This improves robustness to unseen categories without much sacrifice in closed-set performance.
- NNO supports scalable recognition systems that incrementally add classes, detect unseen categories, and minimize operational downtime.The discussion identifies dynamic environments and low downtime as practical deployment settings.
- As known categories increase, OS-NNO remains relatively stable while closed-set performance approaches open-set performance.This pattern motivates the authors’ conjecture that closed-world recognition converges toward open-world recognition as category counts grow.
- Broader evaluation across multiple features and applications remains future work, alongside exploration of improved CAP models and probability calibration.The paper presents NNO as one viable extension while leaving a broader range of approaches open.
7 Supplemental Material : Towards Open World Recognition
The supplemental material tests threshold selection, compares features and datasets, and provides implementation pseudocode for NNO. These experiments examine whether NNO’s observed advantages persist across settings and how thresholding affects recognition performance.
- Threshold selection: Lowering the threshold below τopt significantly reduces the number of correct predictions retained.The selected optimal threshold was approximately τopt = 5000.
- Threshold selection: Threshold selection must balance retaining correct predictions against rejecting unknown categories.The selected τopt maintains this balance in the reported experiments.
- Feature comparisons: Across Dense SIFT, HOG and LBP, the experiments show similar performance trends to those observed on ILSVRC 2010.Absolute performance differs by feature, with Dense SIFT performing best, followed by HOG and LBP.
- Feature comparisons: Closed-set performance of CS-NCM and CS-NNO is comparable, whereas OS-NCM suffers significantly in open-set testing.This pattern is reported across the evaluated feature types.
- Algorithm: The NNO pseudocode normalizes features, learns a metric, estimates τ through cross-class validation, and incrementally updates the model with new class means.The updated model retains the learned metric and threshold while adding the new means.