Source-linked AI summary
DOC: Deep Open Classification of Text Documents
Lei Shu, Hu Xu, Bing Liu
TL;DR
Open classification addresses the closed-world assumption by requiring classifiers to recognize known classes and reject documents from unseen classes. DOC uses deep learning with one-vs-rest sigmoid outputs and Gaussian-fitted rejection boundaries, and the paper reports dramatic improvements over state-of-the-art approaches. The authors identify incremental learning of new classes as future work.
Problem
Dynamic text-classification settings can contain documents from classes absent during training, so classifiers must detect unseen documents rather than assume every test class was seen.
Method
DOC combines a deep CNN-based text classifier with one-vs-rest sigmoids and Gaussian fitting to tighten rejection boundaries and reduce open-space risk.
Results
DOC dramatically outperforms state-of-the-art methods from text and image classification domains, with higher macro-F1 than OpenMax and cbsSVM across reported settings.
Takeaways & Limitations
DOC provides a classifier that handles both seen-class assignment and rejection of unseen documents, and the authors believe it is applicable to images.
Takeaways & Limitations
The paper leaves improved cumulative or incremental learning of new classes without retraining on all past and new data for future work.
Abstract
from arXiv · showhide
Traditional supervised learning makes the closed-world assumption that the classes appeared in the test data must have appeared in training. This also applies to text learning or text classification. As learning is used increasingly in dynamic open environments where some new/test documents may not belong to any of the training classes, identifying these novel documents during classification presents an important problem. This problem is called open-world classification or open classification. This paper proposes a novel deep learning based approach. It outperforms existing state-of-the-art techniques dramatically.
1 Introduction
Open classification addresses the failure of the closed-world assumption in dynamic environments by assigning documents to known classes or rejecting those from unseen classes. DOC is proposed as a deep-learning approach that uses sigmoid-based rejection boundaries and reportedly outperforms existing methods.
- Problem: Dynamic environments violate the closed-world assumption because future documents may contain classes absent from training.Examples include constantly changing social-media topics and new objects encountered by self-driving cars.
- Problem: Open classification requires assigning incoming documents to existing training classes while detecting documents belonging to none of them.The model should identify what it does not know.
- Problem definition: The target model is an (m + 1)-class classifier comprising m seen classes plus a rejection class for unseen documents.Training labels come from the m seen classes, while test instances may be rejected as unseen.
- Prior work: Prior approaches include one-class SVM, center-based similarity learning, open-space-risk methods, and OpenMax, each with stated weaknesses or lower reported performance.Reported issues include weak one-class performance, threshold-selection requirements, and OpenMax assumptions about logits and unseen-class validation data.
- Proposed approach: DOC uses deep learning with a 1-vs-rest sigmoid layer and Gaussian fitting to tighten decision boundaries and reduce open-space risk.Unlike softmax, the sigmoid design supports rejection of documents outside seen-class boundaries.
2 The Proposed DOC Architecture
DOC combines a CNN text-classification architecture with a 1-vs-rest sigmoid output layer and Gaussian-fitted, class-specific thresholds to enable rejection of unseen documents and reduce open space risk.
- 2.1 CNN and Feed Forward Layers of DOC: DOC uses a CNN base with word embeddings, convolution, max-over-time pooling, and fully connected layers that reduce the pooled feature vector to m seen-class dimensions.The intermediate representation h has k dimensions, while d has m dimensions, where m is the number of training classes.
- 2.2 1-vs-Rest Layer of DOC: The final layer contains m 1-vs-rest sigmoid functions, each trained with its class as positive examples and all other training examples as negatives.The model optimizes the summed log loss of the m sigmoid functions on the training data.
- 2.2 1-vs-Rest Layer of DOC: DOC uses sigmoid outputs instead of softmax because softmax normalizes probabilities across seen classes and lacks rejection capability.The 1-vs-rest formulation represents other seen and unseen classes as the rest for each class boundary.
- 2.2 1-vs-Rest Layer of DOC: At test time, DOC rejects a document when every class probability falls below its corresponding threshold; otherwise, it assigns the class with the highest probability.This reinterpretation of the sigmoid outputs prevents multiple predicted labels for one example, unlike multi-label classification.
- 2.3 Reducing Open Space Risk Further: DOC replaces the default threshold ti = 0.5 with a larger boundary when needed, because unseen examples can occupy the gap between the sigmoid boundary and dense positive examples.The desired boundary is di = T with ti ≫ 0.5, which more closely wraps the dense positive examples.
- 2.3 Reducing Open Space Risk Further: For each seen class, DOC fits a half-Gaussian to positive-class probabilities, estimates σi using mirrored points, and sets ti = max(0.5, 1 − ασi).The commonly used α value is 3, and Gaussian fitting allows different classes to have different thresholds.
3 Experimental Evaluation
The evaluation tests DOC on two topic-classification datasets under open-world settings with 25%, 50%, 75%, or 100% of classes used for training. DOC is compared with cbsSVM, OpenMax, and a fixed-threshold DOC variant using macro-F1.
- Datasets: Evaluation uses 20 Newsgroups and 50-class reviews, both treated as topic-classification datasets.The reviews dataset is classified by product topic rather than sentiment.
- Test Settings: Training uses 25%, 50%, 75%, or 100% of classes, while testing includes all classes and unseen examples are rejected.The 100% setting is equivalent to traditional closed-world classification.
- Baselines and Metrics: The comparison includes DOC, cbsSVM, OpenMax, and DOC(t = 0.5), evaluated with macro-F1.cbsSVM and OpenMax are the selected text- and image-classification baselines adapted or compared for this evaluation.
- Results: DOC is markedly better than OpenMax and cbsSVM in macro-F1 across both datasets and all tested class-availability settings.The paper reports especially large advantages at 25% and 50%, when most test examples come from unseen classes.
- Results: Gaussian fitting makes DOC markedly better than DOC(t = 0.5) at 25% and 50%, where many unseen classes are present.At 75%, the fixed threshold is slightly better for 20 Newsgroups but worse for 50-class reviews.
4 Conclusion
The paper concludes that DOC is a deep-learning method for open text classification that performs dramatically better than state-of-the-art methods from text and image classification. Future work targets learning new classes without retraining from scratch and improving testing-time performance.
- Conclusion: DOC is a novel deep-learning method for open text classification that performs dramatically better than state-of-the-art text and image classification methods.The conclusion reports this comparison using the same text datasets and experiment settings.
- Future Work: Future work aims to learn new classes without training on all past and new data from scratch.The stated goal is continual or lifelong learning through improved cumulative or incremental learning.