Source-linked AI summary
Rank consistent ordinal regression for neural networks with application to age estimation
Wenzhi Cao, Vahid Mirjalili, Sebastian Raschka
TL;DR
Ordinal regression methods capture ordered labels through multiple binary classification tasks, but those classifiers can disagree. CORAL imposes theoretical consistency guarantees, extends common CNN architectures, and substantially improves age-estimation performance across three datasets; its experiments focus on age estimation, although the technique is intended for broader ordinal tasks.
Problem
Binary-task ordinal regression can produce inconsistent classifier predictions, undermining the combination of those tasks into an estimated age.
Method
CORAL is a rank-consistent ordinal-regression framework with theoretical classifier-consistency guarantees that can extend common CNN architectures.
Results
CORAL substantially improved CNN predictive performance for age estimation on three independent age-estimation datasets.
Takeaways & Limitations
CORAL provides classifier consistency without requiring rank- or training-label-dependent weighting schemes and can generalize to other ordinal-regression problems and neural-network architectures.
Takeaways & Limitations
The empirical study focuses on age estimation, while broader ordinal-regression applications are proposed rather than evaluated here.
Abstract
from arXiv · showhide
In many real-world prediction tasks, class labels include information about the relative ordering between labels, which is not captured by commonly-used loss functions such as multi-category cross-entropy. Recently, the deep learning community adopted ordinal regression frameworks to take such ordering information into account. Neural networks were equipped with ordinal regression capabilities by transforming ordinal targets into binary classification subtasks. However, this method suffers from inconsistencies among the different binary classifiers. To resolve these inconsistencies, we propose the COnsistent RAnk Logits (CORAL) framework with strong theoretical guarantees for rank-monotonicity and consistent confidence scores. Moreover, the proposed method is architecture-agnostic and can extend arbitrary state-of-the-art deep neural network classifiers for ordinal regression tasks. The empirical evaluation of the proposed rank-consistent method on a range of face-image datasets for age prediction shows a substantial reduction of the prediction error compared to the reference ordinal regression network.
1. Introduction
Ordinal regression uses ordered labels, but neural implementations based on binary subtasks can produce inconsistent classifiers. CORAL addresses this with guaranteed consistency, CNN implementations, and improved age-estimation performance.
- Ordinal regression predicts labels on an ordered scale, unlike classification, while label differences remain arbitrary rather than metric.
- Neural ordinal regression commonly suffers inconsistencies among the binary classifiers used to represent ordinal rankings.Figure 1 contrasts rank-inconsistent predictions with probabilities that decrease consistently.
- CORAL provides theoretical guarantees for classifier consistency and adapts common CNN architectures, including ResNet, to ordinal regression.
- Experiments on face-image age estimation found that CORAL’s guaranteed binary-classifier consistency improves predictive performance over the reference ordinal-regression framework.
- The method is evaluated primarily through age estimation but can also be used for crowd-counting, depth estimation, cell counting, and customer satisfaction.
2. Related work
Prior work reformulated ordinal regression as multiple binary tasks and applied this strategy to CNN-based age estimation. These approaches improved predictive performance but did not guarantee consistent task predictions, motivating CORAL.
- Ordinal-regression research includes proportional odds and hazards models, plus reformulations based on perceptrons, support vector machines, and other classification algorithms.
- OR-CNN transforms K ranks into K −1 binary tasks that predict whether an age exceeds each threshold while sharing intermediate representations.
- OR-CNN achieved state-of-the-art benchmark performance but does not guarantee consistent predictions across its binary classifiers.
- CORAL addresses classifier inconsistency without increasing training complexity, providing a theoretical consistency guarantee.
- Ranking-CNN uses an ensemble of CNNs for binary classification and reported better predictive performance than a single CNN with multiple binary outputs.
- Other CNN approaches include shared lower-layer multi-task models and siamese architectures based on pair-wise comparisons with anchor images.
3. Proposed method
CORAL converts ordinal labels into binary ranking tasks and enforces consistency by sharing weights while learning ordered bias units. Its guarantees cover rank-monotonic predictions under non-negative task weights without cost-matrix convexity or example-specific weighting.
- CNN implementation: CORAL-CNN adapts a CNN’s final output layer to binary ordinal tasks while retaining the surrounding architecture.The paper illustrates this implementation using a ResNet-34-based age-estimation CNN.
- Theoretical guarantees: Unlike the earlier reduction framework, CORAL requires neither convex cost-matrix rows nor example-specific task weights; non-negative fixed task weights suffice.Uniform weighting and dataset-imbalance-aware weighting are both allowed under the stated guarantee.
- Label extension and rank prediction: CORAL extends ordinal labels into K −1 binary tasks, where each task predicts whether an input exceeds rank rk.The binary responses are combined to obtain the predicted rank label.
- Network design: The K −1 binary classifiers share weight parameters but use independent bias units, enforcing ordered decision thresholds.The shared weights horizontally shift the sigmoid curves without intersections, supporting rank-monotonicity.
- Theoretical guarantees: Minimizing CORAL’s weighted cross-entropy yields nonincreasing optimal bias units, satisfying b∗1 ≥ b∗2 ≥ . . . ≥ b∗K−1.The theorem establishes the ordering of the final-layer biases at an optimum.
- Theoretical guarantees: Ordered predicted probabilities produce rank-monotonic binary labels and therefore consistent classifier predictions.The predicted confidence scores decrease across tasks, and the thresholded outputs inherit rank monotonicity.
4. Experiments
Experiments evaluate CORAL for face-image age estimation using three datasets, standardized preprocessing, ResNet-34-based models, and repeated training with fixed evaluation metrics.
- Datasets: Experiments use MORPH-2, CACD, and AFAD face-image datasets for age estimation.The datasets contain 55,608, 159,449, and 165,501 images, respectively.
- Preprocessing: Each dataset is split into 80% training and 20% test data, with images resized and cropped for training or evaluation.Training uses random crops, whereas evaluation uses center crops.
- Models: CORAL-CNN and OR-CNN are implemented by replacing the ResNet-34 output layer with ordinal binary tasks.A standard ResNet-34 with cross-entropy is also used as a baseline in the broader evaluation.
- Evaluation: Performance is evaluated on test sets using mean absolute error (MAE) and root mean squared error (RMSE).The metrics are computed after the last training epoch using the ground-truth and predicted ranks.
- Training: Training uses three random seeds, 200 epochs, adaptive moment estimation, batch size 256, and a tuned learning rate of α = 5 × 10^-5.The best model is selected by validation MAE before independent test evaluation.
5. Results and discussion
Across three age-estimation datasets, CORAL-CNN improves performance over both the cross-entropy baseline and inconsistent ordinal regression, while guaranteeing zero rank inconsistencies.
- Age prediction performance: Across all ordinal regression datasets, CORAL-CNN and OR-CNN outperform the cross-entropy baseline, while CORAL-CNN substantially improves over OR-CNN.The comparisons use ResNet-34-based models and age-prediction errors on the test sets.
- Age prediction performance: Repeated experiments with three random seeds indicate that CORAL-CNN’s improvement over OR-CNN is reproducible rather than coincidental.The same seeds were used across methods for fair comparison.
- Dataset comparison: Overall performance follows MORPH-2 > AFAD > CACD for CE-CNN, CORAL-CNN, and OR-CNN.The paper associates lower performance on AFAD and CACD with lower image quality and, for CACD, a wider age range.
- Rank consistency: OR-CNN has more inconsistencies when its age predictions are incorrect than when they are correct.The authors treat this pattern as evidence that rank inconsistency harms predictive performance.
6. Conclusions
The paper presents CORAL as a theoretically rank-consistent ordinal-regression framework that can extend common neural architectures and improves CNN age-estimation performance.
- Conclusions: CORAL provides theoretical classifier-consistency guarantees without rank- or training-label-dependent weighting schemes.This supports straightforward implementation and efficient model training.
- Conclusions: CORAL substantially improves CNN predictive performance for age estimation on three independent age-estimation datasets.The framework can also be generalized to other ordinal-regression problems and neural architectures.
8. Supplementary Material
The paper derives generalization guarantees for ordinal regression from binary-classification bounds and proves that well-generalizing binary tasks yield a well-generalizing final rank prediction.
- Generalization bounds: The approach derives ordinal-regression generalization bounds from established binary-classification bounds under broad cost-matrix conditions.The required conditions are C_y,rk = 0 when r_k = y and C_y,rk > 0 otherwise.
- Theorem 2: Theorem 2 states that good standard 0/1-loss generalization for every binary task implies good generalization of the final rank prediction.The final rank prediction is produced via h in Eq. 1.
- Proof outline: The proof relates an ordinal prediction error to the binary-task errors between the predicted and true ranks, then takes expectation under P.The intermediate cases cover predictions below or above the true ordinal label.
- Related work: Earlier bounds relied on V-shaped cost-matrix rows and rank-monotonic binary classifiers, with the ordered threshold model as the provided existence case.That model requires convex cost-matrix rows and example-specific task weights.