Do Not Train Away Uncertainty: Early Uncertainty Anchored Calibration

1 INTRODUCTION

• Overconfidence occurs when predicted confidence exceeds actual accuracy, underestimating predictive uncertainty and potentially reducing reliability in high-risk applications.The introduction identifies autonomous driving and medical diagnosis as examples.Source: §1 · p. 1
• Across three models, ECE reaches its minimum early in training or fine-tuning, then rises while accuracy improves only marginally.The authors attribute the later calibration decline to uncertainty awareness being gradually lost.Source: §1 · pp. 2–3
• At ResNet-50 epoch 60, clearer clusters coexist with low-confidence features near their boundaries; by epoch 200, those features nearly disappear as ECE rises.Epoch 60 has lower ECE than epochs 40 and 200, while epoch 200 has the highest accuracy.Source: §1 · p. 2
• EUA-Cal uses the early model as an uncertainty anchor, combining prediction regularization with prototype-structure regularization to preserve uncertainty and mitigate overconfidence.The early model is selected by stopping training at a validation-selected epoch.Source: §1 · pp. 2–3
• EUA-Cal outperforms state-of-the-art calibration methods across diverse deep-network and Transformer architectures.Experiments cover eight models and six datasets.Source: §1 · p. 3

2 RELATED WORK

• DNN calibration approaches include post-hoc adjustment, loss and regularization designs, and mixup-based methods; post-hoc methods can perform poorly under distribution shifts.The section also describes methods tailored to specific network structures, whose cross-structure generalizability remains unclear.Source: §2 · p. 3
• Some calibration methods target specific network structures, but their generalizability across structures remains unclear.The related work also describes EUA-Cal as applicable to fine-tuned LLMs and reports superior performance to compared methods.Source: §2 · p. 3
• For fine-tuned LLMs, methods include MC-Dropout and Deep Ensemble, which require multiple inferences, plus DNN-derived methods such as Mixup and label smoothing.The section also discusses calibration by distillation and IB-EDL.Source: §2 · p. 3

3.1 PROBLEM STATEMENT

• The setup is K-class classification: a model maps each input to logits, then class probabilities, a predicted label, and a confidence.The model is represented as a feature extractor followed by a linear classifier.Source: §3.1 · pp. 3–4

3.2 EARLY UNCERTAINTY ANCHORED CALIBRATION

• EUA-Cal freezes an early model and uses its predictions for EPR and confidence-weighted feature prototypes for PSR.The early checkpoint is selected on validation sets; the two regularizers jointly preserve uncertainty during training or fine-tuning.Source: §3.2 · pp. 4–5, Figure 2 · p. 4
• EPR uses KL divergence between frozen early and current predictions, constraining confidence growth on early-uncertain samples while preserving confident predictions for certain samples.Under vanilla CE, samples identified as uncertain early gradually become highly confident; the method prevents this overconfidence.Source: §3.2 · pp. 4–5
• EUA-Cal’s prototype-prediction ECE keeps decreasing, while vanilla-model prototype ECE reaches its minimum at epoch 60 and rises later.The comparison is for ResNet-50 on CIFAR-10.Source: Figure 3 · p. 5, §3.2 · p. 6
• PSR builds fixed class prototypes from confidence-weighted early features, converts feature-to-prototype similarities into class distributions, and applies KL divergence to current predictions.The frozen prototypes prevent structural targets from drifting during training.Source: §3.2 · pp. 5–6

4.1 EXPERIMENTAL SETUP

• Experiments cover five image-classification models and three LLMs across image-classification and multiple-choice question-answering datasets, evaluating ACC, ECE, and AECE.The setup also evaluates OOD detection using AUROC and FPR95.Source: §4.1 · p. 6

4.2 CALIBRATION RESULTS

• EUA-Cal achieves the lowest ECE and AECE on CIFAR-10, CIFAR-100, and Tiny-ImageNet while maintaining competitive accuracy.It also achieves the best average rank among the compared methods.Source: §4.2 · p. 7, Table 1 · p. 7
• On SciQ, ARC-C, and ARC-E, EUA-Cal has the lowest ECE and AECE, reducing them by 16.24% and 47.92% on average versus each dataset’s second-best method.It also achieves the best average rank; further experiments report consistent improvements across other architectures.Source: §4.2 · p. 7, Table 2 · p. 7

4.3 CALIBRATION UNDER DISTRIBUTION SHIFT

• On CIFAR-10-C and CIFAR-100-C, EUA-Cal has lower ECE than most methods across shift levels; OOD detection improves over vanilla by 1.31% AUROC and 1.79% lower FPR95 on average.The corruption evaluation uses ResNet-50 trained on CIFAR-10 and CIFAR-100.Source: §4.3 · p. 8, Figure 4 · p. 8, Table 4 · p. 9
• When Llama3-8B is fine-tuned on ARC-E and evaluated on ARC-C, SciQ, and CSQA, EUA-Cal reduces AECE by 30.94% on average versus the second-best method.It has the lowest AECE on all three datasets and the lowest ECE on ARC-C and CSQA, ranking second on SciQ.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 FURTHER ANALYSIS

• Combining EPR and PSR gives the lowest ECE on each dataset; EPR reduces ECE across all six datasets, while PSR improves calibration on most.A late anchor can have slightly higher accuracy but substantially worse calibration than the early anchor.Source: §4.4 · pp. 8–9, Table 5 · p. 9
• For ResNet-50 and Llama3-8B, α and β values from [0.1, 0.3] consistently improve calibration; τ is selected by minimizing training-set NLL.The method maintains relatively stable accuracy and competitive calibration across different τ settings.Source: §4.4 · p. 9, Figure 5 · p. 9

• Across models, early models are better calibrated, while continued training often yields modest accuracy gains and worsens calibration.EUA-Cal uses the early model as an uncertainty anchor through EPR and PSR.Source: §5 · p. 10
• EUA-Cal applies EPR and PSR to leverage uncertainty awareness retained by the early model.Source: §5 · p. 10
• Across eight models, experiments on image classification and multiple-choice question answering show improved calibration with competitive accuracy.Source: §5 · p. 10

• Image classification uses CIFAR-10, CIFAR-100, and Tiny-ImageNet; question answering uses SciQ, ARC-C, ARC-E, and CommonsenseQA.Source: §A.1 · p. 14
• CIFAR-10 and CIFAR-100 each contain 60,000 32 × 32 RGB images; each uses 45,000 training, 5,000 validation, and 10,000 test images.The datasets contain 10 and 100 classes, respectively.Source: §A.1 · p. 14
• Tiny-ImageNet has 200 classes and 64 × 64 images, with 90,000 training, 10,000 validation, and 10,000 labeled test images.Source: §A.1 · p. 14
• SciQ contains 13,679 four-choice science questions, using official splits of 11,679/1,000/1,000; questions and choices are input, with the correct answer as label.Source: §A.1 · p. 14
• ARC-C contains challenging grade-school science questions and ARC-E the remainder; their official train/validation/test splits are 1,119/299/1,172 and 2,251/570/2,376.Source: §A.1 · p. 14
• CommonsenseQA has 12,102 five-choice everyday-reasoning questions and official train/development/test splits of 9,741/1,221/1,140; evaluation uses the labeled development set.Test labels are unavailable.Source: §A.1 · p. 14

A.2 TRAINING DETAILS

• Image experiments use five architectures; most train with SGD, while ViT-B/32 uses ImageNet-1K initialization and separate CIFAR settings.CIFAR models train for 200 epochs and Tiny-ImageNet models for 100; ViT-B/32 on CIFAR uses batch size 64, learning rate 0.01 with cosine decay, no weight decay, and gradient clipping norm 1.Source: §A.2 · p. 14
• Llama3-8B, Llama3.1-8B, and Qwen3-8B-Base are fine-tuned with LoRA; adapters use rank 8, scaling 16, dropout 0.1, and learning rate 5 × 10^-5.Training uses cosine annealing, bfloat16 precision, and a maximum token length of 256 for other datasets.Source: §A.2 · p. 14

A.3 HYPERPARAMETER SETTINGS

• Baseline settings include LS α = 0.05, FL γ = 2, and Distillation weight 1.0 with two teachers; the remaining listed methods also receive specified hyperparameters.The appendix specifies additional settings for CRL, MbLS, BalCAL, CSAM, Mixup, MIT, PLP, MaC-Cal, EDL, and IB-EDL.Source: §A.3 · pp. 14–15
• Figure 6 visualizes Llama3-8B and Qwen3-8B-Base SciQ features across fine-tuning steps, with lighter shades marking lower-confidence features.Source: Figure 6 · p. 15
• EUA-Cal uses α and β from [0.1, 0.3], selects τ by minimizing training-sample prototype-prediction NLL, and anchors image models at specified epochs.Image anchors are epoch 60 for CIFAR, epoch 40 for Tiny-ImageNet, and epoch 5 for ViT-B/32; LLM anchors are pretrained models or models after 5–15 fine-tuning steps.Source: §A.3 · p. 15

A.4 EVALUATION PROTOCOLS

• ECE measures the expected difference between accuracy and confidence.It is defined as E[|P(ŷᵢ = yᵢ|p̂ᵢ) − p̂ᵢ|].Source: §A.4 · p. 15
• ECE is a weighted average of the accuracy–confidence difference in each of M equally spaced confidence bins.Each bin’s contribution is weighted by its share of the N samples.Source: §A.4 · p. 15
• AECE uses adaptive confidence bins with equal sample sizes; both calibration metrics use 20 bins and are reported as mean ± standard deviation over three random seeds.Lower ECE and AECE indicate better calibration.Source: §A.4 · p. 15

B VISUALIZATIONS

• Later LLM features form more distinct class clusters, while the number of visible low-confidence samples decreases.The authors interpret this pattern as loss of the early model’s feature uncertainty awareness.Source: B VISUALIZATIONS · p. 16
• Llama3-8B accuracy rises from 93.00% to 93.40%, while ECE rises from 2.82% to 6.33%.The authors associate the pattern with uncertainty awareness in features being lost during later training.Source: B VISUALIZATIONS · p. 16
• Qwen3-8B-Base accuracy rises from 95.30% to 96.70%, while ECE rises from 2.37% to 3.14%.The authors associate the pattern with uncertainty awareness in features being lost during later training.Source: B VISUALIZATIONS · p. 16

C.1 CALIBRATION ON ADDITIONAL MODEL ARCHITECTURE

• EUA-Cal achieves the lowest ECE on every dataset across four additional image models and the best average rank on Llama3.1-8B and Qwen3-8B-Base.It also has the lowest AECE in eight of nine image settings and five of six question-answering settings; on ViT-B/32, ECE falls from 1.92% to 0.48% on CIFAR-10 and from 9.56% to 0.89% on CIFAR-100 versus the vanilla model.Source: §C.1 · p. 16, Table 9 · p. 17, Table 10 · p. 18

C.2 IMPACT OF ANCHOR MODEL SELECTION

• The lowest-ECE anchors are epoch 60 for CIFAR-10 and CIFAR-100 and pretrained Llama3-8B for ARC-C and ARC-E; later anchors raise ECE with only slight accuracy change.Epoch 40 yields lower accuracy, so anchor selection should balance classification and calibration performance rather than simply choose the earliest model.Source: §C.2 · pp. 16–17, Figure 7 · p. 17

C.3 IMPACT OF HYPERPARAMETERS

• Across most α and β settings, EUA-Cal improves calibration on CIFAR-100 and ARC-C while accuracy remains relatively stable.The authors describe this pattern as robustness to these hyperparameters.Source: §C.3 · p. 17, Figure 8 · p. 17
• As τ varies across four datasets, accuracy remains relatively stable while ECE varies more; choosing τ by minimizing training-set NLL gives competitive ECE without a separate τ search.The datasets are CIFAR-10, CIFAR-100, ARC-C, and ARC-E.Source: §C.3 · p. 17, Figure 9 · p. 18