Source-linked AI summary
Two-temperature logistic regression based on the Tsallis divergence
Ehsan Amid, Manfred K. Warmuth, Sriram Srinivasan
TL;DR
Convex surrogate losses can let highly misclassified outliers dominate the objective, motivating more robust classification losses. The paper develops a two-temperature logistic regression loss with tunable boundedness and tail-heaviness, and reports superior performance under label noise across datasets.
Problem
Convex surrogate losses can grow indefinitely on highly misclassified outliers and dominate the objective, motivating study of alternatives for noisy data.
Source: §1 · p. 1Method
The paper defines a multiclass logistic surrogate over linear activations using two temperatures, with a Tsallis-divergence motivation and separate control of loss boundedness and probability tail-heaviness.
Source: §1 · pp. 1–2Results
At 10% noise, 2TLR had superior performance to LR and t-LR in all cases on all datasets, with optimal temperatures often satisfying 0 < t1 < 1.
Source: §7.2 · p. 8Takeaways & Limitations
The authors conclude that tuning non-convexity and boundedness is crucial for robustness to instance and label noise, with computation time comparable to logistic regression.
Source: §8 · p. 8The empirical-loss minimization described implicitly assumes class labels are sampled from the tempered conditional distribution.
Source: §5 · p. 5
Abstract
from arXiv · showhide
We develop a variant of multiclass logistic regression that is significantly more robust to noise. The algorithm has one weight vector per class and the surrogate loss is a function of the linear activations (one per class). The surrogate loss of an example with linear activation vector $\mathbf{a}$ and class $c$ has the form $-\log_{t_1} \exp_{t_2} (a_c - G_{t_2}(\mathbf{a}))$ where the two temperatures $t_1$ and $t_2$ ''temper'' the $\log$ and $\exp$, respectively, and $G_{t_2}(\mathbf{a})$ is a scalar value that generalizes the log-partition function. We motivate this loss using the Tsallis divergence. Our method allows transitioning between non-convex and convex losses by the choice of the temperature parameters. As the temperature $t_1$ of the logarithm becomes smaller than the temperature $t_2$ of the exponential, the surrogate loss becomes ''quasi convex''. Various tunings of the temperatures recover previous methods and tuning the degree of non-convexity is crucial in the experiments. In particular, quasi-convexity and boundedness of the loss provide significant robustness to the outliers. We explain this by showing that $t_1 < 1$ caps the surrogate loss and $t_2 >1$ makes the predictive distribution have a heavy tail. We show that the surrogate loss is Bayes-consistent, even in the non-convex case. Additionally, we provide efficient iterative algorithms for calculating the log-partition value only in a few number of iterations. Our compelling experimental results on large real-world datasets show the advantage of using the two-temperature variant in the noisy as well as the noise free case.
1 Introduction
The paper introduces two-temperature logistic regression to control loss boundedness and probability tail-heaviness, addressing the poor noise robustness of convex surrogate losses. It also studies the loss’s properties and reports Bayes consistency, including in the non-convex case.
1 Introduction
- Convex surrogate losses can grow indefinitely on highly misclassified outliers, allowing them to dominate the objective and weakening robustness to noise.The paper motivates introducing non-convexity to bend down the loss on such points.Source: §1 · pp. 1–2
- The two-temperature loss is upper-bounded at 2.5 for t1 = 0.6 and t2 = 1.6, whereas the cited t-logistic loss diverges as the margin tends to negative infinity.The general cap is 1/(1 − t1).Source: Figure 1 · p. 2
- The paper argues that tail-heaviness alone is insufficient for handling outliers and label noise, while controlling loss boundedness is also crucial for robustness.The two temperatures allow transitions between convex and more robust quasi-convex losses.Source: §1 · p. 2
- Two-temperature logistic regression uses separate temperatures to control loss boundedness and the tail-heaviness of the predictive probabilities.The loss is defined over the linear activation vector for an example and its class label.Source: §1 · pp. 1–2
- The paper provides iterative normalization algorithms, analyzes temperature-dependent loss properties and properness, and establishes Bayes consistency even for non-convex losses.Properness is achieved by switching to the optimizer’s escort probability.Source: §1 · p. 2
2 Tsallis Entropy and Tsallis Divergence
This section introduces temperature-parameterized logarithm and exponential functions and the associated Tsallis entropy and divergence. These generalizations retain familiar functions in the limit t → 1, while the t-exponential has heavier tails for t > 1.
2 Tsallis Entropy and Tsallis Divergence
- The log_t function generalizes the standard logarithm, is increasing, and recovers it as t → 1, but does not generally satisfy the logarithm product rule.For 0 < t < 1 it is lower-bounded by −1/(1 − t); for t > 1 it is upper-bounded by 1/(t − 1).Source: §2 · pp. 2–3
- Tsallis entropy and Tsallis divergence generalize Shannon entropy and KL divergence, respectively, and recover them as t → 1.The divergence is defined between probability distributions p(x) and q(x).Source: §2 · p. 3
- The inverse function exp_t has heavier tails than the standard exponential for t > 1, and defines a t-exponential family normalized by G_t.G_t is the convex log-partition function that ensures the distribution sums to one.Source: §2 · p. 3
3 Two-temperature Logistic Regression
Two-temperature logistic regression models class probabilities with a t2-exponential distribution and defines its surrogate loss using a Tsallis divergence with temperature t1. The temperatures affect boundedness and gradient weighting, while an iterative procedure computes the normalization value when needed.
3 Two-temperature Logistic Regression
- The class probabilities use a t2-exponential model, which recovers softmax when t2 = 1, and G_t2 ensures they sum to one.For t2 ≠ 1, G_t2 generally lacks a closed-form solution, so the paper provides an iterative computation method.Source: §3 · p. 3
- Because G_t2 generally has no closed form for t2 ≠ 1, the paper supplies an iterative algorithm for computing it efficiently.The method takes activations and a temperature t > 1 as input.Source: §3 · pp. 3–4
- The surrogate loss compares the empirical label distribution with the predicted t2-exponential probabilities using a Tsallis divergence with temperature t1.The algorithm using this loss is named Two-Temperature Logistic Regression (2TLR).Source: §3 · pp. 3–4
- For 0 < t1 < 1, each observation’s loss is capped at 1/(1 − t1), and when t2 > t1, its gradient factor dampens observations with small predicted probabilities.The loss is bounded only when 0 < t1 < 1; the importance factor vanishes when t1 = t2.Source: §3 · p. 4
4 Binary Classification
The binary formulation represents two class scores with a single margin and defines probabilities using the generalized log-partition function. Temperature choices determine whether the loss is convex or quasi-convex.
4 Binary Classification
- For two classes labeled ±1, the model uses separate class weights and expresses the activations through a single margin.The binary parameterization uses W = [w+, w−] and margin w⊤x.Source: §4 · p. 4
- The generalized log-partition function normalizes the two t2-exponential probabilities and is invariant to a shared shift of the activations.For t2 = 1, the probabilities reduce to logistic probabilities; for t2 ≠ 1, the binary partition value has no closed-form solution and is computed iteratively.Source: §4 · pp. 4–5
- The binary loss recovers standard logistic regression when t1 = t2 = 1 and t-logistic loss when t1 = 1 and t2 = t > 1.The t2 = 1 probability case also recovers logistic probabilities.Source: §4 · p. 4
4.1 Properties
- The loss is convex when t1 ≥ t2 and t1 ≥ 1, with curvature increasing as t1 − t2 grows; it is quasi-convex when t1 < t2 or t1 < 1.The latter condition applies for any t2 ≥ 0 when t1 < 1.Source: §4.1 · p. 5
5 Implications of Using the Tsallis Divergence
The paper examines the implications of fitting predictions with the Tsallis divergence. Minimizing the resulting observed-label objective entails a tempered sampling assumption, and the optimizer's escort probabilities recover the conditional probabilities.
5 Implications of Using the Tsallis Divergence
- Minimizing the observed-label objective implicitly assumes that class samples are drawn from a tempered conditional distribution.The derivation uses a Monte Carlo approximation of the expected Tsallis divergence and removes terms corresponding to unobserved labels.Source: §5 · p. 5
- The loss is not proper because its predicted probabilities differ from the true posterior, but the escort probabilities at the optimum match the correct conditional probabilities.The minimizing prediction is proportional to the posterior raised to a temperature-dependent power.Source: §5 · p. 5
- Table 1 covers classification accuracy with 10% label noise under random, small-margin, and large-margin selection of points.Small-margin selects points with the smallest margins; large-margin selects those with the largest margins.Source: Table 1 · p. 6
6 Bayes-consistency
The paper establishes Bayes-consistency for the multiclass surrogate loss by characterizing its minimizing scores. Although those scores correspond to a powered posterior, their argmax preserves the most probable class.
6 Bayes-consistency
- Bayes-consistency requires a unique minimizing margin vector whose largest component identifies a class with maximal posterior probability.The definition applies for all possible label-probability distributions.Source: §6 · p. 6
- At the optimum, the predicted probabilities are proportional to the true class posterior raised to the power 1/t1.The minimizer is unique under the stated margin constraint.Source: §6 · p. 6
- The multiclass surrogate is Bayes-consistent because taking the argmax is invariant to raising posterior probabilities to a positive power.Thus the powered-posterior form of the optimum preserves the class decision.Source: §6 · p. 7
7 Experiments
The experiments compare 2TLR with logistic, hinge, and t-logistic losses on binary classification tasks, including instance noise, label noise, and runtime. They report stronger noise robustness for 2TLR, with runtime generally comparable to other methods.
7 Experiments
- Experiments compare 2TLR, LR, hinge loss, and t-LR across four datasets; 10% of instances are used for testing, with 10-fold cross-validation on the remainder to choose parameters and results averaged over 10 random train-test splits.The datasets are Fashion MNIST, CIFAR-10, Character Font Images, and Covertype.Source: §7 · p. 7
7.1 Instance Noise
- Up to 33% instance noise, 2TLR performance is not considerably affected, and it is more robust than the other methods.The authors attribute this robustness to capping the surrogate loss; t-LR sometimes performs worse than LR.Source: §7.1 · p. 7
7.2 Label Noise
- At 10% label noise, 2TLR consistently outperforms the other methods across all cases and datasets.Most optimal settings use 0 < t1 < 1, though some coincide with LR or t-LR settings.Source: §7.2 · p. 8
7.3 Runtime
- 2TLR runtime is generally comparable to the other methods and is sometimes faster than LR, but it takes considerably longer on Covertype.The overhead of calculating Gt is reported as negligible.Source: §7.3 · p. 8
8 Conclusions
The paper concludes that 2TLR uses two temperatures to tune loss properties and that tuning them is crucial to robustness against instance and label noise. Its computation time is comparable to logistic regression.
8 Conclusions
- The first temperature tunes loss non-convexity and boundedness, while the second controls probability tail-heaviness.Source: §8 · p. 8
- Experiments indicate that tuning non-convexity and boundedness is crucial for robustness to both instance and label noise.Source: §8 · p. 8
- The computation time of 2TLR is comparable to logistic regression.Source: §8 · p. 8
A Verification of Iterative Algorithms for Computing Gt
The appendix verifies convergence of the iterative algorithm for computing Gt, first in the binary case and then by extension in the multiclass case.
A Verification of Iterative Algorithms for Computing Gt
- The binary-case proof establishes convergence of the iterates to the value that normalizes the sum.Source: A Verification of Iterative Algorithms… · p. 10
- The iterates form a monotonically decreasing sequence bounded below by the target value.The proof uses induction and the increase of Z as the iterate decreases.Source: A Verification of Iterative Algorithms… · p. 10
- Continuity of Z(u)^(1−t) yields convergence of the iterates to the target value.Source: A Verification of Iterative Algorithms… · p. 10
B Proof of Remark 1
The proof characterizes convex and quasi-convex regions of the surrogate loss under different temperature relationships.
B Proof of Remark 1
- The loss is convex when t2 = t1 ≥ 1 and also when t1 ≥ 1 and t1 ≥ t2.Source: B Proof of Remark 1 · p. 10
- For t2 ≥ t1, the proof identifies an inflection point using a condition involving the predicted probability and λ.Source: B Proof of Remark 1 · pp. 10–11
- When t2 < 1, the loss is convex above the stated activation threshold and constant below it, making it quasi-convex.Source: B Proof of Remark 1 · p. 11