Source-linked AI summary
LSHTC: A Benchmark for Large-Scale Text Classification
Ioannis Partalas, Aris Kosmopoulos, Nicolas Baskiotis, Thierry Artieres, George Paliouras, Eric Gaussier, Ion Androutsopoulos, Massih-Reza Amini, Patrick Galinari
TL;DR
Large-scale text classification must handle tens of thousands to hundreds of thousands of classes, complex hierarchies, sparsity, and imbalance. The paper presents LSHTC benchmarks and challenge tracks built from Wikipedia- and ODP-related corpora, with evaluation measures and online comparison infrastructure. Across challenge editions, flat and hierarchical systems were both competitive, while winning approaches varied across tasks.
Problem
Large-scale hierarchical classification requires learning with very many, imbalanced, statistically dependent categories that conventional learning frameworks do not adequately handle.
Method
The paper constructs LSHTC benchmark datasets from Wikipedia, DBpedia, and ODP sources, organizes tracks across scales and task settings, and provides evaluation measures and comparison infrastructure.
Results
Across LSHTC editions, flat and hierarchical systems were both competitive, with winning approaches varying across tasks and the strongest systems often achieving close results.
Takeaways & Limitations
The released benchmarks and evaluation infrastructure provide reference datasets and comparisons intended to support further research in large-scale classification.
Abstract
from arXiv · showhide
LSHTC is a series of challenges which aims to assess the performance of classification systems in large-scale classification in a a large number of classes (up to hundreds of thousands). This paper describes the dataset that have been released along the LSHTC series. The paper details the construction of the datsets and the design of the tracks as well as the evaluation measures that we implemented and a quick overview of the results. All of these datasets are available online and runs may still be submitted on the online server of the challenges.
1 Introduction
Large-scale and hierarchical text classification exposes challenges that conventional learning frameworks do not handle well, especially as the number and complexity of categories grow. LSHTC addresses this need through challenges spanning large class counts and multiple classification settings, using Wikipedia and ODP corpora.
- Large datasets, increasing complexity, and tens of thousands of classes create classification problems beyond conventional learning frameworks.
- Hierarchical classification is challenging because growing hierarchies add classes alongside documents, preserve sparsity, create severe imbalance, and encode statistical dependencies.
- Scaling algorithms to very large class counts is a central challenge; DMOZ contains over one million categories.
- Classification systems must also account for complex relationships among categories, such as the relationships linking more than 20,000 Wikipedia categories.
- LSHTC evaluates large-scale hierarchical classification through tracks ranging from thousands to hundreds of thousands of classes, including standard, multi-task, and unsupervised settings.
- The initiative builds its tracks from Wikipedia and ODP corpora and follows related large-scale classification challenges and workshops.
2 LSHTC Dataset
The LSHTC datasets combine DBpedia and DMOZ sources with sparse vector representations, category mappings, and hierarchy files for large-scale classification tracks. Across challenge iterations, the datasets expand from single-label tree classification to multi-label and non-tree settings, with track-specific preprocessing and splits.
- Data sources: LSHTC datasets originate from DBpedia and DMOZ, whose instances use content vectors, description vectors, or both.Content vectors index web pages, while description vectors index manually created ODP descriptions.
- Data representation: Each instance is stored as a sparse vector with a category label followed by feature-frequency pairs.In multi-label data, comma-separated category numbers represent multiple labels; feature id 0 is reserved for internal indexing and ignored during classification.
- Dataset organization: Datasets provide track-specific token and category mappings, training/validation/test splits, and hierarchy files describing parent-child relations.Test labels are replaced by category 0, while hierarchy files encode paths or individual parent-child relations depending on the challenge.
- LSHTC1: The first LSHTC challenge used single-label DMOZ data with a tree hierarchy of maximum depth 5 and allowed classification only to leaf nodes.Instances deeper than depth 5 were assigned to their depth-5 ancestor, and four tracks combined content and description vectors in different ways.
- Dataset statistics: The paper reports dataset statistics for the first LSHTC and LSHTC2, including main and dry-run datasets and class distributions.The supplied material identifies the corresponding tables and figures but does not include their numerical entries.
- LSHTC2 and later iterations: LSHTC2 introduced multi-label instances and non-tree hierarchies, using all DMOZ content vectors and DBpedia tracks that varied in size and graph structure.DBpedia Track 2 contains fewer instances and categories with a cleaned DAG, whereas Track 3 retains cycles; later iterations reused these datasets and added unprocessed medium DBpedia text.
3 Evaluation
LSHTC evaluates large-scale classifiers with both flat measures, which judge predictions as correct or incorrect, and hierarchical measures, which recognize graded correctness through class relationships. Across challenge editions, leading flat and hierarchical systems were often competitive, while the strongest approaches varied by track.
- Evaluation measures: Flat measures classify predictions as correct or wrong, whereas hierarchical measures account for relationships between predicted and gold labels and allow partial degrees of error.The challenges used both evaluation types across classification tracks.
- Evaluation measures: Single-label flat evaluation uses accuracy, precision, recall, and F1, with macro precision and recall averaged across categories.These measures are computed from true-positive, true-negative, false-positive, and false-negative counts.
- Evaluation measures: Multi-label evaluation includes micro-F1 and derives micro-precision and micro-recall from class-level true-positive, false-positive, and false-negative counts.The implemented multi-label measures follow versions presented in prior work.
- Evaluation measures: Statistical significance tests can assess whether two systems differ significantly under a flat evaluation measure.The paper identifies S-tests and p-tests for this purpose.
- Challenge results: In LSHTC1, hierarchical and flat systems were the two best approaches, while centroid methods also performed very well.The hierarchical system used polynomial SVMs, and the flat system used online training techniques.
- Challenge results: In LSHTC2, flat systems won different tasks, although top-tier systems using hierarchical and flat strategies achieved very close results.The winning approaches used an associative network with score post-processing and BM25-based KNN with thresholding.
- Challenge results: In LSHTC3, flat and hierarchical approaches were competitive, and the first-track winner used hierarchical meta-learning with thresholding.The method extracted classifier scores as meta-features and classification accuracy as a meta-label.
4 Conclusions
The paper presents LSHTC as a benchmark for assessing classification algorithms with hundreds of thousands of target classes. Its downloadable datasets and evaluation oracles are intended to support reproducible comparison and continued research in large-scale classification.
- Conclusions: LSHTC assesses classification algorithms in settings containing hundreds of thousands of target classes.The challenge ran from 2009 to 2014.
- Conclusions: The challenge benchmarks and evaluation oracles are available online for downloading, method evaluation, and comparison with participating systems.The authors describe these resources as supporting their long-term goal of advancing large-scale classification research.