Source-linked AI summary
CORDIAL: Calibrating Ordinal LLM Outputs from Few Labels
Xiangwei Wang, Peng Wang, Saman Halgamuge
TL;DR
LLM distributions over ordered labels can be systematically biased or misshapen, so they are noisy readings of true labels rather than calibrated statistics. CORDIAL corrects these readings with a five-parameter Bayesian channel that preserves ordinal order. Across the reported reader and label-budget settings, it generally outperforms competing calibrators, while its posterior also supports learned priors and reader fusion.
Problem
LLM outputs over ordered labels can be biased, over- or under-dispersed, and poorly calibrated, even when the model understands the text.
Source: §1 · p. 1Method
CORDIAL treats an LLM's ordinal distribution as a noisy measurement and calibrates it with a five-parameter Bayesian channel that preserves first-order stochastic order.
Source: §2.1 · p. 1, §2.3 · p. 2Results
At every 5–100-label budget across four benchmarks, the 7B reader's CORDIAL has the lowest NLL among nine calibrators; other readers show the same advantage in 56 of 60 settings.
Source: §3.1 · pp. 3–4Takeaways & Limitations
The five-parameter posterior supports priors learned across tasks, fusion of multiple readers, and per-user adaptation.
Source: §1 · p. 1, §3.2 · p. 4, §4 · p. 4The crossover formula predicts its scale rather than bounding it, because higher-order terms and shrinkage shift the actual crossover.
Source: §2.6 · p. 2
Abstract
from arXiv · showhide
A large language model (LLM) can turn a text into a distribution over an ordered scale, but that distribution is a noisy measurement: saturated, compressed or exaggerated, and biased in a consistent direction. We propose CORDIAL, which treats the model's output as a noisy reading of the true label and corrects it with a channel of five interpretable parameters. The channel is small enough for its posterior to be averaged from a handful of labels, and we prove that the resulting calibration preserves first-order stochastic order. On Amazon reviews and CMU-MOSEI transcripts with four LLMs, CORDIAL has the lowest log loss among nine calibrators in 76 of 80 settings with 5 to 100 labels; with 20 labels and the main 7B reader, it matches the strongest baseline using 28-54 labels. The same posterior lets us learn priors from other tasks and fuse several LLMs. Unrestricted calibrators such as Dirichlet calibration overtake it only as the calibration set grows into the hundreds or thousands.
1. INTRODUCTION
LLM distributions over ordered labels can be systematically biased or too wide or narrow, making them noisy measurements rather than calibrated statistics. CORDIAL models those errors with a five-parameter Bayesian channel and uses its posterior for ordinal calibration and related applications.
1. INTRODUCTION
- LLM readings can be systematically too generous, too spread out or too narrow, and less trustworthy than their confidence suggests.The paper frames calibration as estimating a channel from few labels rather than collecting more labels.Source: §1 · p. 1
- A scalar reliability cannot move probability mass to a neighboring class, while unrestricted matrices and Dirichlet calibration use many parameters for few labels.CORDIAL instead uses five parameters—temperature, offset, gain, concentration, and strength—in a structured semantic channel.Source: §1 · p. 1
- The five-parameter posterior can be averaged to hedge against estimation error with a few dozen labels, and the channel can be stacked with logit-space calibrators.The channel has mixture and unimodal-location forms.Source: §1 · p. 1
- The contributions include an ordinal-order-preserving Bayesian channel, a predicted crossover scale for unrestricted calibrators, and posterior-based applications including learned priors, multi-reader fusion, and per-user shifts.Evidence covers Amazon Reviews 2023 and CMU-MOSEI transcripts with four readers.Source: §1 · p. 1
2. CORDIAL: STRUCTURED CHANNELS
CORDIAL treats frozen LLM outputs as noisy measurements of ordinal labels and estimates a structured channel from a small calibration set. Its five-parameter channel adjusts ordinal errors while preserving stochastic order, and its posterior also supports stacking and cross-task, multi-reader, and per-user extensions.
2.1. LLM outputs as noisy ordinal measurements
- CORDIAL estimates a row-stochastic channel that maps a frozen reader’s noisy ordinal distribution to a calibrated distribution over the true label.The reader stays frozen, and estimating the channel from a few labeled texts is the calibration problem.Source: §2.1 · pp. 1–2
2.2. Why a scalar reliability is not enough
- A scalar reliability only rescales a reading’s excess over the prior, so it cannot move probability mass to a neighboring class to correct directional error.For a saturated reading one class too high, the scalar channel assigns that class no more probability than ignoring the reader.Source: §2.2 · p. 2
2.3. An affine channel with interpretable parameters
- CORDIAL’s five parameters—temperature, offset, gain, concentration, and strength—model ordinal errors while preserving first-order stochastic order for both channel forms.The mixture form retains the reading’s shape; the location form is unimodal, suited to labels that are rounded averages.Source: §2.3 · p. 2
2.4. Bayesian estimation from few labels
- The five-parameter posterior is averaged over 300 importance-reweighted draws from a Laplace approximation at the mode.The paper uses independent Gaussian priors centered at a near-identity channel.Source: §2.4 · p. 2
2.5. Stacking with logit-space calibrators
- CORDIAL channels are stacked with temperature scaling, vector scaling, and proportional-odds regression using weights selected by out-of-fold log-likelihood.With fewer than eight labels, the stack uses equal weights.Source: §2.5 · p. 2
2.6. When does structure pay?
- Structure is expected to win below n⋆ = (du − ds)/(2δ), where δ is the structured family’s approximation gap; higher-order terms and shrinkage shift the actual crossover.The expression predicts its scale rather than bounding it.Source: §2.6 · p. 2
2.7. What a five-parameter posterior enables
- The posterior supports learning channel priors across tasks, combining several frozen LLMs, and shifting predictions for repeated texts from one author.The multi-reader channel has 2R + 3 parameters; personalization uses a per-user offset rather than reliability rescaling.Source: §2.7 · p. 3
3. EXPERIMENTS
The experiments compare calibrators across Amazon Reviews and CMU-MOSEI, finding CORDIAL strongest at small label budgets for the 7B reader. The posterior is also evaluated for learned priors, multi-reader fusion, and user adaptation.
3. EXPERIMENTS
- Amazon Reviews uses three 1–5-star domains, while CMU-MOSEI uses seven sentiment levels; calibrators share label budgets from 5 to 2,000 and are scored by NLL and RPS.The study evaluates four frozen readers and averages over 20 label-subset draws.Source: §3 · p. 3
3.1. Few labels: structure beats size
- With the 7B reader, CORDIAL has the lowest NLL among nine calibrators from 5 to 100 labels on all four benchmarks, and at 20 labels matches the logit stack with 28–54 labels and Dirichlet calibration with 79–132.Fig. 1 plots test NLL relative to CORDIAL against the number of calibration labels.Source: §3.1 · p. 3, Figure 1 · p. 4
- CORDIAL is best in 56 of 60 settings with 5–100 labels across the 3B, 14B, and Llama readers; in-context examples yield 2.3–3.3 times higher NLL than calibration.The in-context examples have accuracy within −0.06 to +0.02 of CORDIAL.Source: §3.1 · p. 4
3.2. What the posterior buys
- The prior learned from other datasets gains an average of 0.017, 0.016, 0.011, 0.006, and 0.003 nats at 5, 10, 20, 50, and 100 labels over twelve Amazon reader–domain pairs.The gains are resolved for 9 or 10 pairs at each budget, with no resolved loss.Source: §3.2 · p. 4
- Four-reader fusion beats the 7B reader alone in all 28 settings from 5 to 500 labels, by 0.008–0.090 nats, and beats fusion without channels in 27 settings by 0.011–0.043 nats at 20 labels.On MOSEI with 10–20 labels, Llama alone is 0.018–0.030 better and multi-reader fusion adds at most 0.006.Source: §3.2 · p. 4
- After eleven or more earlier user texts, a per-user shift channel gains 0.100, 0.046, 0.044, and 0.048 nats on Music, Beauty, Software, and MOSEI, respectively.Across Amazon texts, it beats per-user reliability, temperature, and random-intercept ordinal regression; on MOSEI it ties random-intercept regression.Source: §3.2 · p. 4
4. CONCLUSION
CORDIAL treats an LLM’s ordinal output as a noisy sensor with systematic errors and calibrates those errors with a five-parameter Bayesian channel. The posterior extends to learned priors, multiple readers, and user-specific adaptation.
4. CONCLUSION
- An LLM’s ordinal output can be treated as a noisy sensor with systematic offset, gain, and concentration errors.The conclusion frames these as systematic distortions of the reading.Source: §4 · p. 4
- A five-parameter Bayesian channel calibrates those distortions from few labels.The channel naturally extends to learned priors, multiple readers, and user-specific adaptation.Source: §4 · p. 4
- The same posterior supports learned priors, multiple-reader fusion, and user-specific adaptation.Source: §4 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The study reuses Amazon Reviews 2023 and CMU-MOSEI transcripts with their provided user and speaker identifiers. It involves no new participant recruitment or intervention.