Source-linked AI summary
Open-world Learning and Application to Product Classification
Hu Xu, Bing Liu, Lei Shu, P. Yu
TL;DR
Open-world learning must recognize seen classes while rejecting unseen ones and accommodating new classes, a combination existing approaches handle with limitations. L2AC uses a meta-classifier and examples from a dynamically maintained set of seen classes for classification and rejection. On product classification, the authors report superior performance to strong baselines.
Problem
Open-world learning requires classifying seen classes, rejecting unseen classes, and adding new classes, while existing approaches have limitations in supporting these capabilities together.
Source: §1 · p. 1Method
L2AC trains a meta-classifier on other classes, then compares test examples with examples from each maintained seen class to classify or reject them.
Source: §2 · p. 2, §2.2 · p. 4Results
Takeaways & Limitations
New classes can be incrementally accepted by adding their examples to the seen-class set, without retraining the meta-classifier.
Source: §5 · p. 6The meta-classifier is trained on other classes, and broad coverage of features in those classes is desirable for seen and unseen classes at test time.
Source: §2.2 · p. 4
Abstract
from arXiv · showhide
Classic supervised learning makes the closed-world assumption, meaning that classes seen in testing must have been seen in training. However, in the dynamic world, new or unseen class examples may appear constantly. A model working in such an environment must be able to reject unseen classes (not seen or used in training). If enough data is collected for the unseen classes, the system should incrementally learn to accept/classify them. This learning paradigm is called open-world learning (OWL). Existing OWL methods all need some form of re-training to accept or include the new classes in the overall model. In this paper, we propose a meta-learning approach to the problem. Its key novelty is that it only needs to train a meta-classifier, which can then continually accept new classes when they have enough labeled data for the meta-classifier to use, and also detect/reject future unseen classes. No re-training of the meta-classifier or a new overall classifier covering all old and new classes is needed. In testing, the method only uses the examples of the seen classes (including the newly added classes) on-the-fly for classification and rejection. Experimental results demonstrate the effectiveness of the new approach.
1 INTRODUCTION
Open-world learning must classify known classes, reject unseen ones, and accommodate newly learned classes without retraining. L2AC uses a meta-classifier to support both recognition and rejection while maintaining a dynamic set of seen classes.
1 INTRODUCTION
- Existing approaches commonly require retraining or incremental training to add classes, while incremental-learning methods lack unseen-class rejection.The paper frames these as limitations of different existing approaches to open-world learning.Source: §1 · p. 1
- Open-world learning requires classifying examples from seen classes, rejecting unseen classes, and incorporating new classes without retraining.These are the two challenges the paper identifies: classification with unseen-class rejection, and incremental addition of classes when enough data is available.Source: §1 · pp. 1–2
- L2AC maintains a dynamic set of seen classes, allowing new classes to be added or deleted without retraining the meta-classifier.A new class can be accepted by adding the class and its examples to the set.Source: §1 · p. 2
- L2AC compares a test example with examples from each seen class and uses a learned meta-classifier for classification or rejection.The meta-classifier is trained on other classes; if no seen class is accepted, the test example is rejected as unseen.Source: §1 · p. 2
- The proposed approach is evaluated on product classification, where the reported results show competitive performance.Source: §1 · p. 2
2 L2AC FRAMEWORK
L2AC combines a ranker with a meta-classifier to classify examples against a dynamically maintained set of seen classes and reject examples that match none. The meta-classifier compares a test example with retrieved class examples, aggregates their similarities, and is trained on separate meta-training classes.
2 L2AC FRAMEWORK
- L2AC compares a test example with each seen class and rejects it if no class probability exceeds 0.5; otherwise, it predicts the highest-probability class.The threshold is fixed rather than tuned for particular classes, and the class set can be dynamically maintained.Source: §2 · p. 2, §2.1.3 · pp. 3–4
- For each seen class, the ranker retrieves the top-k nearest examples, which the meta-classifier evaluates alongside the test example.The ranker supports retrieval; the learned metric space, voting, and rejection are handled by the meta-classifier.Source: §2 · p. 2
2.1.1 Example Representation and Memory.
- The meta-classifier stores continuous example representations in memory so the retrieved top-k examples can be looked up by index.The memory is a matrix E ∈ R^n×|h|, where n is the number of seen-class examples and |h| is the hidden dimension size.Source: §2.1.1 · pp. 2–3
2.1.2 1-vs-many Matching Layer.
- The 1-vs-many matching layer computes an individual similarity score between the test example and each of a seen class’s top-k retrieved examples.The matching network transforms each pair into a similarity space using element-wise absolute difference and sum, then passes the concatenation through fully connected layers and a sigmoid.Source: §2.1.2 · p. 3
2.1.3 Open-world Learning via Aggregation Layer.
- An aggregation layer combines the k similarity scores into one probability for whether the test example belongs to the seen class.The aggregation layer uses a BiLSTM as a parametric voting mechanism over the retrieved examples.Source: §2.1.3 · p. 3
2.2 Training of Meta-Classifier
- The meta-classifier is trained on examples from a separate set of meta-training classes, with positive pairs from the same class and negative pairs from other classes.For each training example, the method retains the top-n negative classes and weights positive and negative pairs in an n:1 ratio; separate validation classes support model selection and tuning k and n.Source: §2.2 · p. 4
3 EXPERIMENTS
The experiments evaluate L2AC on Amazon product categories, test its ranker and hyperparameters, and compare it with open-world classification baselines. L2AC outperforms DOC, especially with fewer seen classes, while retaining an advantage as classes are added.
3 EXPERIMENTS
- The experiments address how k and n affect L2AC and how it compares with open-world text classifiers that require retraining.Table 1 reports weighted and macro F1 across settings with 25, 50, and 75 seen classes.Source: §3 · p. 4, Table 1 · p. 6
3.1 Dataset
- The Amazon dataset uses 1,000 encoder-training classes, 1,000 meta-training classes, and 100 test classes, with 50 held-out test examples per test class.Product categories are paths to leaf nodes; the study removes products assigned to multiple classes.Source: §3.1 · p. 4
3.2 Ranker
- The ranker selects top-n negative classes by cosine similarity between each class’s averaged encoded representation and a meta-training example.For test examples, cosine similarity ranks examples within each seen class.Source: §3.2 · p. 4
3.3 Evaluation
- Evaluation uses macro F1 and weighted F1 across seen classes and the rejection class, with weighted F1 accounting for class example counts.The models are trained with 10 initializations and results are averaged; parallel ranker implementation is left for future work.Source: §3.3 · pp. 4–5
3.4 Hyper-parameters
- Validation selects k = 5 and n = 9, achieving 87.60% weighted F1; larger n values yield limited gains while increasing training data and time.For k = 5, n = 14 yields 87.69% and n = 19 yields 87.68%.Source: §3.4 · p. 5
3.5 Compared Methods
- The comparison includes DOC variants with different encoders and rejection thresholds, alongside L2AC ablations for voting, similarity functions, k, and n.DOC is retrained from scratch for each seen-class set, while L2AC uses the seen examples without training on them.Source: §3.5 · p. 5
3.6 Results Analysis
- L2AC outperforms DOC, especially with fewer seen classes, and remains better as the seen-class set grows, though its performance gradually drops.Testing with 25 seen classes emphasizes rejection; testing with 75 emphasizes seen-class classification.Source: §3.6 · p. 6, Table 1 · p. 6
- Single-similarity L2AC variants perform worse; non-parametric top-3 voting may not improve performance and can introduce higher variance.The results also indicate that using multiple nearest examples and negative classes is effective.Source: §3.6 · p. 6
4 RELATED WORK
Prior open-world approaches commonly add rejection to closed-world classifiers, while incremental-learning methods add classes but still require training. The paper positions L2AC as a meta-learning approach to product classification, a setting not previously studied as open-world learning in the cited work.
4 RELATED WORK
- Open-world methods commonly add thresholds to closed-world classifiers for rejection, but adding classes requires full or incremental retraining.Class-incremental methods such as iCaRL also train when each new class is added.Source: §4 · p. 6
- The paper frames L2AC as meta-learning similarity between an example and an arbitrary class, an approach the authors say they had not seen in open-world learning.The authors relate the framework to meta-learning.Source: §4 · p. 6
- Although product classification has been studied largely in hierarchical settings, the cited work had not treated it as open-world learning despite dynamic taxonomies.Open-world learning is also called open-set recognition in text mining and computer vision.Source: §4 · p. 6
5 CONCLUSIONS
The paper presents L2AC, a meta-learning framework for open-world learning applied to product classification. It reports that L2AC can add classes without retraining and outperforms strong baselines.
5 CONCLUSIONS
- L2AC is a meta-learning framework for open-world learning, applied to product classification.Source: §5 · p. 6
- L2AC accepts new classes by adding their examples without retraining, and learns rejection rather than relying on empirically set thresholds.Source: §5 · p. 7
- The experiments report superior performance to strong baselines.Source: §5 · p. 7