The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling

Yaoshiang Ho; Samuel Wookey

Source-linked AI summary

The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling

Yaoshiang Ho, Samuel Wookey

arXiv:2001.00570v1Jan 3, 2020cs.LGcs.AIstat.ML

TL;DR

Standard classifier metrics and losses do not directly encode unequal real-world consequences or selected probabilistic mislabeling costs. The paper proposes Real World Cost and RWWCE for binary and single-label multiclass training, reports reductions in costly errors and Real World Cost in MNIST scenarios, and sketches an MLE-based proof.

Problem
Accuracy and F1 do not weight classifier errors by real-world impact, while softmax cross-entropy does not directly penalize probabilistic false negatives during training.
Method
The paper introduces Real World Cost and the RWWCE loss function, which uses real-world cost weights for binary and single-label multiclass classification.
Results
RWWCE reduces specific mislabeling errors and Real World Cost in the paper’s MNIST scenarios, while increasing top-1 error in the reported experiments.
Takeaways & Limitations
Real World Cost and RWWCE provide a training and evaluation framework centered on the differing costs of false negatives, false positives, and selected multiclass mistakes.

Abstract

from arXiv · show

In this paper, we propose a new metric to measure goodness-of-fit for classifiers, the Real World Cost function. This metric factors in information about a real world problem, such as financial impact, that other measures like accuracy or F1 do not. This metric is also more directly interpretable for users. To optimize for this metric, we introduce the Real-World- Weight Crossentropy loss function, in both binary and single-label classification variants. Both variants allow direct input of real world costs as weights. For single-label, multicategory classification, our loss function also allows direct penalization of probabilistic false positives, weighted by label, during the training of a machine learning model. We compare the design of our loss function to the binary crossentropy and categorical crossentropy functions, as well as their weighted variants, to discuss the potential for improvement in handling a variety of known shortcomings of machine learning, ranging from imbalanced classes to medical diagnostic error to reinforcement of social bias. We create scenarios that emulate those issues using the MNIST data set and demonstrate empirical results of our new loss function. Finally, we sketch a proof of this function based on Maximum Likelihood Estimation and discuss future directions.

I. INTRODUCTION

The paper addresses shortcomings of standard classifier metrics and training objectives when error costs are unequal, classes are imbalanced, or mistakes carry social and medical consequences. It proposes Real World Cost and the RWWCE loss function, then applies the approach to binary and single-label multiclass settings.

A 1% disease prevalence can make a classifier that never predicts disease 99% accurate, motivating alternatives to accuracy and F1.The paper also discusses recall, precision, oversampling, undersampling, and weighted labels as existing responses.
Softmax cross-entropy does not directly penalize probabilistic false negatives during training, limiting responses to social bias and medical diagnostic errors.
Real World Cost weights each classification error by estimated real-world impact, such as financial cost, making the metric more interpretable for some users.
RWWCE is designed to optimize Real World Cost, generalizing binary and softmax cross-entropy while adding weights for false positives.
The proposed weighting framework derives weights from dataset or problem costs and treats them as fixed real-world quantities rather than repeatedly tuned hyperparameters.
The paper evaluates RWWCE on binary imbalanced data and single-label multiclass data, sketches a Maximum Likelihood Estimation proof, and discusses future directions.

II. RELATED WORK

Related work covers weighted losses, resampling, F1 optimization, bias-reduction methods, and cost-sensitive multiclass classification. These approaches provide context for RWWCE’s use of weights to address imbalance and specific errors.

Weighted binary cross-entropy can increase or decrease the penalty for probabilistic false negatives, while weighted categorical cross-entropy changes class-level penalties.
Focal Loss weights examples by class and classification difficulty, assigning less weight to easy examples.
Prior work describes weighting specific target–predicted class combinations in single-label multiclass classification.
Oversampling replicates minority examples, undersampling removes majority examples, and SMOTE interpolates between existing minority-class points.
Social-bias research includes modifying word representations, regulating undesirable behavior, and ignoring low-confidence model outputs.
Weighted maximum likelihood estimators address imbalanced classes from a Maximum Likelihood Estimation perspective, which the paper extends in its proof sketch.
Some work directly optimizes F1, while threshold tuning after accuracy training remains a two-step alternative that does not adapt model weights to F1.

III. REAL-WORLD-WEIGHT CROSS-ENTROPY LOSS FUNCTION

RWWCE modifies cross-entropy by incorporating real-world costs for false negatives and false positives. Its binary and single-label categorical variants support cost-sensitive training, including label-specific probabilistic false-positive penalties.

Neural-network loss compares predicted probabilities with target labels and supplies penalties whose derivatives adjust trainable weights during backpropagation.
The binary RWWCE first term penalizes probabilistic false negatives; for target 1 and output 0.6, the unweighted penalty is -log(0.6) = 0.22.
The corresponding second term applies the same logic to probabilistic false positives.
Standard weighted binary cross-entropy adjusts the importance of positive labels, commonly giving minority classes more weight.
RWWCE assigns one weight to missing a positive and a separate weight to missing a negative in binary classification.
For single-label categorical classification, RWWCE adds weights for false negatives and a matrix of label-specific false-positive costs.
The multilabel, multiclass extension remains undeveloped; its false-positive matrix could be as large as 2^K by 2^K.

IV. BINARY CLASSIFICATION OF IMBALANCED

The binary experiment uses many MNIST-derived imbalanced datasets to compare standard, threshold-tuned, and RWWCE-trained classifiers under explicitly unequal false-negative and false-positive costs. RWWCE reduces Real World Cost despite increasing top-1 error.

100 MNIST datasets were created by treating batches of one digit as positive and the remaining 63,000 nonmatching examples as negative.The datasets covered digits “0” through “9” and used 67.5% training, 7.5% validation, and 25% test splits.
100 control and 100 experimental neural networks differed only in their loss function, with standard binary cross-entropy used for controls and RWWCE for experiments.
The comparison also threshold-tuned control outputs across all possible thresholds to maximize F1 without changing network weights.
The experiment set the marginal cost of a false negative at 2,000 and the marginal cost of a false positive at 100.
RWWCE produced fewer false negatives but more false positives than both controls, increasing top-1 error while lowering Real World Cost.The paper attributes this pattern to false positives being much less costly than false negatives.

A. COMPARING REAL WORLD COST TO F1

The paper argues that Real World Cost can be preferable to F1 because it incorporates real-world outcome costs and directly aligns training with the target measure.

The authors argue that optimizing for Real World Cost can be superior to F1-based approaches.
Optimizing accuracy first and F1 later through threshold search is a two-step process that does not adapt neural-network weights directly to F1.
Real World Cost incorporates the costs and benefits of real-world outcomes, whereas F1 does not factor in this information.
F1 is mathematically focused on infrequent positives and can be high for a trivial classifier when positives are frequent.

B. COMPARING RWWCE TO HEURISTICS IN BINARY

For binary classification, RWWCE is equivalent to weighted binary cross-entropy but adds a framework for setting weights from estimated real-world costs rather than heuristic tuning.

RWWCE is mathematically equivalent to weighted binary cross-entropy for binary classification.
RWWCE directly applies costs to false positives and false negatives, while weighted binary cross-entropy uses one parameter for false negatives.
The main binary-classification contribution is a framework for deciding RWWCE’s two weights.
The weights should reflect estimated real-world marginal costs, such as $2,000 for a false negative and $100 for a false positive.
Oversampling adds computational complexity, whereas undersampling loses training information and introduces another hyperparameter to estimate and tune.

V. FALSE NEGATIVES IN SINGLE-LABEL, MULTICLASS

The multiclass experiment tests whether RWWCE reduces specifically costly mislabelings by assigning extra penalty to selected false-negative/false-positive label pairs.

The experiment used MNIST with 67.5% training, 7.5% validation, and 25% test data.
90 experimental neural networks covered all possible high-cost false-negative/false-positive label combinations.
Each experimental model assigned cost 19 to one selected label pair while leaving other false-negative costs at 1.
RWWCE reduced mislabeled digits and Real World Cost but increased top-1 error.
The increase in top-1 error was expected because training minimized Real World Cost rather than maximizing overall accuracy.
The reported table caption gives a mean number of high-cost false-negative/false-positive cases of 6.67.

VI. SOFTMAX

The paper identifies a limitation of categorical cross-entropy under softmax: it penalizes probabilistic false negatives but not probabilistic false positives, which RWWCE is designed to address.

Softmax outputs sum to 1, with each output interpreted as the probability or confidence of a class.
Categorical cross-entropy penalizes confidence missing from the target class but does not penalize probability assigned to incorrect classes.
RWWCE is designed to increase penalties for specific probabilistic false negatives associated with specific target labels.
In binary classification, binary cross-entropy typically uses one output for positive probability and imputes negatives through the summed expression.
A cited multilabel result found categorical cross-entropy performed better than binary cross-entropy when individual labels were highly imbalanced.

VII. CONNECTION TO MAXIMUM LIKELIHOOD

The paper connects RWWCE training to Maximum Likelihood Estimation by treating costly real-world outcomes as weighted or imputed observations. In a coin-flip example, this yields p=0.9 and the corresponding 9:1 sticker-to-toy ratio.

Connection to Maximum Likelihood: The derivation rewrites the weighted loss as a likelihood objective under an i.i.d. training-set assumption.The paper represents target values as repeated observations according to their weights.
Binary example: A binary RWWCE example assigns costs of 9 to false negatives and 1 to false positives when predicting coin flips.The training set contains one head and one tail, while p is the classifier’s sole trainable output.
Binary example: Gradient descent minimizes the loss near p=0.9, and the paper proves that this is the minimum.The cost function is evaluated as a function of p.
Binary example: The value p=0.9 implies a 9:1 ratio of stickers to toys because 1-p=0.1.The ratio reflects the game’s reward values for heads and tails.
Connection to Maximum Likelihood: For this example, binary RWWCE gradient descent is equivalent to Maximum Likelihood Estimation on imputed real-world outcomes.The imputed outcomes are stickers and toys derived from observed heads and tails and their associated costs.
Limitations and future work: The Maximum Likelihood analysis is limited to an input-free binary classifier, with input-conditional multiclass analysis deferred to future work.The paper also identifies future directions beyond this proof setting.