PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities

Xintong Shi, Wenzhi Cao, Sebastian Raschka

arXiv:2111.08851v5cs.LGcs.CVstat.ML

TL;DR

Conventional classification losses ignore the ordering of ordinal targets, while CORAL’s rank consistency relies on weight sharing that may restrict network expressiveness. CORN uses conditional training sets and the chain rule to obtain rank consistency without that constraint; experiments report better performance than CORAL on the evaluated datasets, with a tie against OR-NN on Fireman.

  • Problem

    Conventional multicategory classification losses ignore ordinal target order, while CORAL achieves rank consistency through a weight-sharing constraint that may limit network expressiveness.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    CORN uses conditional training sets and the chain rule for conditional probability distributions to achieve rank consistency without CORAL’s weight-sharing constraint.

    Source: §1 · p. 2

  • Results

    CORN outperforms other binary label extension methods on MORPH-2 and AFAD and ties OR-NN on Fireman; on Fireman, OR-NN and CORN outperform CE-NN and CORAL in test MAE and test RMSE.

    Source: §5 · pp. 8–9

  • Takeaways & Limitations

    Experiments report CORN’s improved predictive performance over CORAL when training convolutional and fully connected neural architectures on ordinal response variables.

    Source: §6 · p. 9

    An exhaustive study of other neural-network ordinal-regression methods is outside the paper’s scope.

    Source: §2.3 · p. 3

Abstract

from arXiv · show

In recent times, deep neural networks achieved outstanding predictive performance on various classification and pattern recognition tasks. However, many real-world prediction problems have ordinal response variables, and this ordering information is ignored by conventional classification losses such as the multi-category cross-entropy. Ordinal regression methods for deep neural networks address this. One such method is the CORAL method, which is based on an earlier binary label extension framework and achieves rank consistency among its output layer tasks by imposing a weight-sharing constraint. However, while earlier experiments showed that CORAL's rank consistency is beneficial for performance, it is limited by a weight-sharing constraint in a neural network's fully connected output layer, which may restrict the expressiveness and capacity of a network trained using CORAL. We propose a new method for rank-consistent ordinal regression without this limitation. Our rank-consistent ordinal regression framework (CORN) achieves rank consistency by a novel training scheme. This training scheme uses conditional training sets to obtain the unconditional rank probabilities through applying the chain rule for conditional probability distributions. Experiments on various datasets demonstrate the efficacy of the proposed method to utilize the ordinal target information, and the absence of the weight-sharing restriction improves the performance substantially compared to the CORAL reference approach. Additionally, the suggested CORN method is not tied to any specific architecture and can be utilized with any deep neural network classifier to train it for ordinal regression tasks.

1 Introduction

Ordinal targets have an intrinsic order that conventional multicategory classification losses ignore. CORN is introduced as a rank-consistent ordinal-regression framework that avoids CORAL’s weight-sharing constraint and improves predictive performance over that reference.

1 Introduction

  • Conventional multicategory losses ignore ordinal order, treating predictions with different rank distances alike even though distances between ordinal ranks cannot be quantified.The introduction contrasts severe-versus-none and severe-versus-moderate errors, while noting ordinal ranks do not have measurable distances as in metric regression.Source: §1 · p. 1
  • CORN achieves rank consistency through a new framework based on the chain rule for conditional probability distributions.Its training scheme uses conditional training sets to obtain unconditional rank probabilities.Source: §1 · p. 2
  • CORN guarantees rank consistency without CORAL’s weight-sharing constraint, which can limit neural-network expressiveness, and without substantially increasing computational complexity.CORAL uses identical output-node weight parameters, differing only in bias units.Source: §1 · p. 2
  • Experiments across neural-network architectures and datasets report improved predictive performance over the more restrictive CORAL reference framework.The paper presents the method as applicable across different neural-network architectures.Source: §1 · p. 2

2 Related Work

The related work traces extended-binary ordinal regression from a general reduction framework to neural methods that address rank consistency. It also distinguishes other neural approaches and states that an exhaustive comparison of them is outside this paper’s scope.

2.1 Ordinal Regression Based on Extended Binary Classification Subtasks

  • Extended-binary ordinal regression converts ranks to binary vectors, trains binary classifiers, and computes a predicted rank; Li and Lin unified these approaches in a reduction framework.Related approaches were developed for perceptrons and support vector machines.Source: §2.1 · p. 2

2.2 Addressing Rank Consistency in Neural Networks for Ordinal Regression

  • OR-NN adapts the extended-binary framework to deep networks but has rank inconsistencies; CORAL addresses these with shared output-layer weights, potentially limiting expressiveness.The paper investigates rank consistency without CORAL’s weight-sharing constraint or increased training complexity.Source: §2.2 · pp. 2–3

2.3 Other Neural Network-Based Methods for Ordinal Regression

  • Other neural approaches include ordinal forests, soft ordinal labels, distance-metric learning [24], differentiable sorting, and ranking constraints.An exhaustive study of these methods is outside the paper’s scope; supplementary comparisons include SORD and CNNPOR.Source: §2.3 · p. 3

3 Proposed Method

CORN addresses rank inconsistency in binary-extension ordinal regression without CORAL’s weight-sharing constraint. It estimates conditional probabilities on conditional training subsets, then applies the chain rule to obtain rank-consistent outputs.

3.1 Preliminaries x[i], y[i] N

  • Ordinal regression maps inputs to labels drawn from an ordered rank set and seeks to minimize a loss function.The paper denotes the inputs as x[i], labels as y[i], and the mapping as h.Source: §3.1 · p. 3

3.2 Motivation

  • CORAL’s shared output weights may limit a network’s flexibility and expressiveness, motivating a rank-consistent approach without that constraint.CORAL uses the same weights for all binary tasks, which differ only in their biases.Source: §3.2 · pp. 3–4

3.3 Rank-consistent Ordinal Regression based on Conditional Probabilities

  • CORN estimates conditional probabilities and applies the chain rule to guarantee rank consistency across its binary tasks.Unlike CORAL, it does not require a weight-sharing constraint.Source: §3.3 · pp. 4–5

3.4 Conditional Training Subsets

  • CORN constructs conditional training subsets so each binary task estimates a conditional probability on data satisfying the corresponding condition.For example, the passage describes estimating a conditional probability using only examples whose labels exceed the first threshold.Source: §3.4 · p. 5

3.6 Rank Prediction

  • At inference, CORN thresholds the predicted probabilities for its K−1 binary tasks and sums the resulting labels to obtain a rank index.The predicted rank is r_q.Source: §3.6 · p. 6

4 Experiments

The experiments use balanced versions of image and tabular datasets, compare CORN with ordinal-regression baselines, and evaluate test-set rank prediction errors after validation-based tuning.

4.1 Datasets and Preprocessing

  • The study uses balanced MORPH-2 and AFAD age-label subsets, an unbalanced AES beauty-score dataset, and a balanced 16-category Fireman dataset.MORPH-2 has 20,625 images and 33 labels; AFAD has 13 labels; AES scores are averaged and rounded to labels from 1 to 5.Source: §4.1 · p. 6

4.2.1 Comparison with binary label extension frameworks for ordinal regression

  • CORN is compared with OR-NN, CORAL, and cross-entropy classifiers using ResNet34 on image datasets and an MLP on Fireman.The methods share similar backbone architectures and require only small output-layer changes; CORN uses a binary conditional-probability task layer.Source: §4.2.1 · p. 7

4.3 Training and Evaluation

  • Models are evaluated by test-set MAE and RMSE, with hyperparameters and the best training epoch selected using validation performance.Each method is trained five times with different random seeds after grid search over the same hyperparameter ranges.Source: §4.3 · p. 7

4.4 Hardware and Software

  • The paper provides its experiment source code and a user-friendly CORN implementation in the coral-pytorch package.Experiments used PyTorch 1.8 on RTX 2080Ti and T4 graphics cards.Source: §4.4 · p. 8

5 Results and Discussion

CORN outperforms the other binary label extension methods on MORPH-2 and AFAD and ties OR-NN on Fireman. The ablation indicates that training subsets support meaningful rank probabilities and usually improve predictive performance.

5 Results and Discussion

  • CORN outperforms the other binary label extension methods on MORPH-2 and AFAD and ties OR-NN on Fireman.Table 3 reports test prediction errors averaged across five random-seed runs, with standard deviations; the image datasets use ResNet34 and Fireman uses an MLP.Source: §5 · p. 8, Table 3 · p. 9
  • Although CORAL's rank consistency was beneficial in earlier work, it performs worse than OR-NN on balanced MORPH-2 and AFAD, where CORN performs better than OR-NN.The authors suggest CORAL's output-layer weight-sharing constraint could limit network expressiveness.Source: §5 · p. 8
  • On Fireman, OR-NN and CORN have identical test MAE and RMSE and outperform CE-NN and CORAL on both metrics.The paper describes this 16-category prediction task as relatively easy for a fully connected network regardless of loss function.Source: §5 · p. 9

5.1 Ablation Study

  • The no-subset CORN variant gives each task at least as many training examples as regular CORN, but its conditional probability scores are described as meaningless despite retaining rank consistency.The modified variant trains each task on the full training batch rather than a subset.Source: §5.1 · p. 9
  • Training subsets support meaningful, theoretically justified rank probabilities and improve predictive performance on every dataset except MORPH-2.Table 4 reports MAE test errors for balanced datasets using ResNet34.Source: §5.1 · p. 9, Table 4 · p. 9

6 Conclusions

CORN provides rank-consistent ordinal regression through conditional training datasets and improves predictive performance over CORAL. The method was used with convolutional and fully connected architectures and is presented as applicable to other datasets and neural network structures.

6 Conclusions

  • CORN improved predictive performance over the rank-consistent reference framework CORAL.Source: §6 · p. 9
  • The framework trains convolutional and fully connected neural architectures on ordinal response variables.Source: §6 · p. 9
  • CORN is developed for rank-consistent ordinal regression using conditional training datasets.Source: §6 · p. 9
  • Although experiments focused on image and tabular data, the authors describe CORN as applicable to other datasets and neural network structures.Source: §6 · p. 10

8 Supplementary Material

The supplementary material supports CORN's probability-estimation rationale and provides additional comparisons across image and text datasets. It also documents implementation details, including a numerically stable loss formulation and figures explaining the method.

8.1 Theoretical Analysis of Conditional Probability Estimation

  • The theoretical analysis interprets minimizing the CORN loss as maximum-likelihood estimation of conditional probabilities, then uses the probability chain rule to obtain unconditional rank probabilities.The authors state that conditional training sets are needed for the loss to have that conditional-probability maximum-likelihood interpretation.Source: §8.1 · p. 12

8.2 Generalization Bounds

  • Theorem 1 bounds rank-prediction generalization error using binary-task errors and states that CORN generalizes well when those binary tasks generalize well.Source: §8.2 · p. 12

8.3 Comparison with Other Deep Learning Methods for Ordinal Regression

  • On AES, CORN outperforms CNNPOR and SORD overall with a pre-trained VGG16 backbone.Table S2 reports test-set prediction errors and identifies best results in bold; MAE is lower-is-better.Source: §8.3 · p. 13, Table S2 · p. 14

8.4 Additional Results on Text Datasets using Recurrent Neural Networks

  • On TripAdvisor and Coursera, CORN outperforms the other methods on test MAE, while CORAL performs better on test RMSE.The authors report that both methods show reliable performance over the text datasets.Source: §8.4 · p. 15, Table S4 · p. 15

8.5 Detailed Performance Table

  • Table S5 gives a more detailed version of the main-paper results, listing performance for each individual random seed.Its caption describes test-set prediction errors for image and tabular datasets with balanced class labels.Source: §8.5 · p. 16, Table S5 · p. 16

8.6 Numerically Stable Loss Function

  • The alternative loss formulation enables use of logsigmoid rather than log(1-sigmoid(z)), yielding more numerically stable gradients during backpropagation.The logits are the net inputs of the last layer.Source: §8.6 · p. 17

8.7 Additional Figures Explaining the CORN Method

  • The supplementary figures visualize CORN loss computation with conditional training subsets and show an architecture using k − 1 output nodes instead of k.The hidden layers can use an existing backbone such as ResNet34.Source: Figure S2 · p. 19, Figure S3 · p. 20
Loading 2111.08851v5…