Source-linked AI summary
A neural network approach to ordinal regression
Jianlin Cheng
TL;DR
Ordinal regression requires learning ordered categories while retaining the discrete nature of classification. The paper presents NNRank, a multilayer neural-network generalization of the perceptron, and reports better benchmark performance than classification neural networks and comparable performance to support vector machines and Gaussian processes.
Problem
Ordinal regression must learn finite ordered categories, combining properties of classification and metric regression while accounting for category order.
Method
NNRank adapts traditional multilayer neural networks for ordinal regression by generalizing perceptron learning and estimating ordinal-category probabilities with multiple outputs.
Results
NNRank outperforms standard classification neural networks and achieves comparable performance to state-of-the-art support vector machine and Gaussian-process ordinal-regression methods on benchmark datasets.
Takeaways & Limitations
NNRank provides a complementary tool for large-scale ordinal tasks because it supports online and batch learning, very large datasets, and rapid predictions.
Takeaways & Limitations
Further work is needed to ensure monotonic decrease of neural-network outputs and to derive general error bounds under the binary-classification framework.
Abstract
from arXiv · showhide
Ordinal regression is an important type of learning, which has properties of both classification and regression. Here we describe a simple and effective approach to adapt a traditional neural network to learn ordinal categories. Our approach is a generalization of the perceptron method for ordinal regression. On several benchmark datasets, our method (NNRank) outperforms a neural network classification method. Compared with the ordinal regression methods using Gaussian processes and support vector machines, NNRank achieves comparable performance. Moreover, NNRank has the advantages of traditional neural networks: learning in both online and batch modes, handling very large training datasets, and making rapid predictions. These features make NNRank a useful and complementary tool for large-scale data processing tasks such as information retrieval, web page ranking, collaborative filtering, and protein ranking in Bioinformatics.
1. Introduction
Ordinal regression assigns instances to finite ordered categories, combining discrete classification with ranking structure. NNRank adapts multilayer neural networks to this task, offering nonlinear modeling, scalable training, and rapid prediction while outperforming standard neural-network classification on benchmarks.
- Ordinal regression assigns data points to finite, ordered categories, unlike classification because category order matters and unlike metric regression because responses are discrete.
- Existing ordinal-regression methods include perceptrons, neural networks, Gaussian processes, and support vector machines, commonly using pairwise constraints or multiple thresholds.
- Large-margin pairwise approaches can create O(n^2) ranking constraints, limiting use on medium-sized datasets and potentially overfitting noise from incomparable pairs.
- Support-vector and Gaussian-process approaches can face slow prediction or difficulty handling large training datasets, constraining their use for time-critical or large-scale tasks.
- NNRank generalizes perceptron learning to multilayer neural networks, uses multiple outputs for ordinal-category probabilities, and supports online or batch learning, nonlinear data, large datasets, and rapid prediction.
- NNRank outperforms standard classification neural networks on the same benchmark datasets while achieving comparable performance to support-vector-machine and Gaussian-process ordinal-regression methods.
2. Method
NNRank adapts a multilayer neural network to ordinal categories by encoding each category cumulatively, preserving category order while retaining standard neural-network training.
- Ordinal formulation: NNRank encodes category k with ones for categories 1 through k and zeros thereafter, imposing the ordinal structure on outputs.Its output vector can be interpreted as a cumulative probability distribution over ordered categories.
- Network architecture: The network uses d input nodes, K output nodes, and optionally one or more hidden layers to learn ordinal relations.The described implementation uses one hidden layer in a standard two-layer feedforward network.
- Output layer: Unlike traditional softmax outputs, each NNRank output node uses an independent sigmoid to estimate whether the input belongs to category i.The first k outputs target 1 and the remaining outputs target 0 for an example in category k.
- Learning: Training minimizes relative entropy or squared error between target and output vectors, then propagates errors backward through the network using gradient descent.The experiments use squared error with standard back-propagation; output derivatives differ because the transfer function is an independent sigmoid.
- Learning: NNRank supports online updates per example and batch updates per group of examples, as in traditional neural-network training.The method changes the output transfer function and derivative while keeping training otherwise very similar to standard neural networks.
- Prediction: At prediction time, the method scans outputs in order and returns the last index whose output exceeds a predefined threshold T.With T set to 0.5, scanning stops at the first output below the threshold or after all outputs are examined.
3. Experiments and Results
NNRank was evaluated on eight benchmark datasets using repeated train/test splits and ensemble predictions, outperforming NNClass in nearly all cases and matching state-of-the-art ordinal methods overall.
- Benchmark Data and Evaluation: Eight standard ordinal-regression datasets were evaluated using 20 independent train/test partitions.The datasets were originally metric-regression benchmarks whose targets were discretized into five ordinal categories.
- Experimental Setup: The experiments tuned hidden units, epochs, and learning rate, then averaged predictions from five models initialized with different weights.Training used online learning, with learning-rate halving when training errors rose continuously for a predefined number of epochs.
- Comparison with Neural Network Classification: NNRank outperformed NNClass in both mean zero-one and mean absolute error on all but one dataset.Mean zero-one error was about 4% lower on Stock and Pyrimidines, while mean absolute error fell by about .05 on four datasets.
- Comparison with Gaussian Processes and Support Vector Machines: NNRank achieved the best zero-one-error results on Diabetes, Triazines, and Abalone, while different baselines led on the remaining datasets.The comparison included SVM, GP-MAP, and GP-EP.
- Comparison with Gaussian Processes and Support Vector Machines: NNRank achieved the best mean absolute error on Diabetes and Abalone, with GP-EP, GP-MAP, and SVM leading on other datasets.The distribution of best results varies across datasets rather than favoring one method universally.
- Overall Findings: Across the eight datasets, NNRank performed comparably to the three state-of-the-art ordinal-regression methods.Large-scale bioinformatics evaluation was still in progress.
4. Discussion and Future Work
NNRank adapts multilayer neural networks to ordinal regression by generalizing the perceptron approach, retaining flexible training and prediction properties while leaving further improvements open.
- Contribution: NNRank adapts traditional neural networks to ordinal regression as a multilayer generalization of the one-layer perceptron approach.The method is described as simple and novel within the paper.
- Neural Network Advantages: NNRank supports both online and batch learning, enabling adaptive learning in real time.Its multilayer structure and nonlinear transfer function provide stronger fitting ability than perceptron methods.
- Scalability and Usability: Neural networks can be trained iteratively on very large datasets, although training is more complex than with support vector machines and Gaussian processes.The training process remains familiar to users of traditional neural networks.
- Applications: Rapid prediction after training makes NNRank suitable for time-critical and large-scale ranking tasks.The paper identifies information retrieval, web-page ranking, collaborative filtering, and bioinformatics as relevant application areas.
- Future Work: Future work includes monotonic output transfer functions, general error bounds, and other multilayer perceptron implementations.These directions are presented as ways to further improve neural-network ordinal regression.