ProbPlug: A Plugin Uncertainty Network for Reliable Confidence in LLM Binary Classification

• LLMs perform well on textual and multimodal classification but can make incorrect predictions, and their confidence estimates are often unreliable for risk-sensitive applications.Source: §1 · pp. 1–2
• Existing confidence-estimation approaches include verbalization, logit-based methods, self-consistency, trained probes, and hybrids, but are not specifically designed for LLM-based classification.The paper therefore focuses on confidence estimation for LLM-based binary classification.Source: §1 · p. 2
• ProbPlug hypothesizes that LLM decision-making relates to representations across Transformer layers, drawing inspiration from confidence associated with successive stages of human decision-making.Source: §1 · p. 2
• ProbPlug is a lightweight plug-and-play framework that predicts confidence from hidden representations of a frozen LLM without altering the base model.It focuses on binary classification with Yes/No outputs and uses only newly generated tokens as input.Source: §1 · p. 2
• Across multiple benchmarks, ProbPlug provides better-calibrated confidence estimates, improves downstream classification performance with negligible inference overhead, and shows promising cross-task generalization.Source: §1 · p. 2

2 Problem Statement

• Prompted binary classification outputs such as “Yes” or “No” indicate a decision but do not naturally provide calibrated confidence for reliability-sensitive use.The generated token does not explicitly reflect confidence.Source: §2 · p. 3
• ProbPlug estimates the probability that a generated answer is correct using hidden representations of its final token across transformer layers.The layer-wise representations capture complementary information at different stages of model computation.Source: §2 · p. 3
• A lightweight external network estimates decision confidence from a frozen LLM, making ProbPlug plug-and-play and computationally efficient.Here, Y = 1 denotes that the generated answer is correct.Source: §2 · p. 3

3.1 Model Architecture

• ProbPlug combines Token Compression, Layer Information Aggregation, and a classification head while keeping the backbone LLM frozen.The framework aggregates layer-wise hidden signals for confidence estimation.Source: §3.1 · p. 3

Hidden State Extraction

• Selecting the final generated token’s hidden state across layers captures the model’s decision and empirically improves cross-task generalization.The default hidden-state tensor has dimensions (N, S, L, D), with S = 1.Source: Hidden State Extraction · pp. 3–4

Token Compression Module

• A token-wise nonlinear transformation and two fully connected layers refine token features and compress them into one representation per layer.The fully connected layers map token dimension S to 2S and then to 1.Source: Token Compression Module · p. 4

Layer Information Aggregation Module

• Multi-head attention uses a learnable query to adaptively weight and fuse layer-wise representations, which are then passed to a lightweight classifier.Different transformer layers contribute unequally to decision confidence.Source: Layer Information Aggregation Module · p. 5, Confidence Prediction · p. 5

Confidence Prediction

• The classifier’s output P represents the estimated probability that the LLM’s generated answer is correct.The output is computed using learnable parameters Wc and bc.Source: Confidence Prediction · p. 6

3.2 Cross-task Generalization and Multi-class Classification

• For unseen tasks, the confidence estimator can be reused by modifying the prompt; multi-class tasks use one-vs-rest decisions and softmax-normalized class confidence scores.This formulation extends ProbPlug to multi-class and multimodal classification scenarios.Source: §3.2 · p. 6

Dataset

• The evaluation spans five public text datasets covering spam detection, sentiment analysis, toxicity detection, and product review classification.The datasets are SMS Spam Collection, SST-2, Toxic Comment Classification, Civil Comments, and Amazon Polarity.Source: Dataset · p. 6
• IEMOCAP provides a multimodal evaluation set with 5,531 utterances across four emotion categories.It is used to evaluate performance on multimodal large models.Source: Dataset · p. 6
• Qwen3-8B is the baseline model, applied directly to binary text classification across all datasets.Source: Baselines · p. 7
• The comparison includes Qwen output variants using Verbalization or Logit.Source: Baselines · p. 7
• Advanced inference strategies Self-Consistency and CISC are also evaluated.Source: Baselines · p. 7
• SAPLMA is evaluated as a method that attaches and trains a three-layer MLP classification head on the models’ internal representations.Source: Baselines · p. 7
• Model performance is quantified using F1-score and area under the precision-recall curve (AUPRC).The passage describes these metrics as common in related literature [19].Source: Baselines · p. 7

4.2 Main Performance

• ProbPlug outperforms competitive confidence-estimation methods on same-task F1-score and AUPRC, and state-of-the-art approaches across most cross-task metrics.The reported results cover in-task and cross-task datasets and are averaged over multiple runs.Source: §4.2 · p. 7
• Without additional training, ProbPlug is evaluated on cross-task datasets by modifying prompts and outperforms current state-of-the-art approaches across most metrics.Source: §4.2 · p. 7
• On SST2, ProbPlug achieves a slightly lower F1-score than CISC.Source: §4.2 · p. 7
• ProbPlug requires one inference pass, whereas CISC requires multiple repeated inferences.The paper identifies this as an inherent efficiency advantage over CISC.Source: §4.2 · p. 7
• On SST2, ProbPlug performs better than CISC in AUPRC, which the authors say could enable higher performance through threshold adjustment.Source: §4.2 · p. 7
• Across in-task and cross-task datasets, the reported results are averaged over multiple runs.Source: §4.2 · p. 7

4.3 Multi-class Classification Performance

• ProbPlug shows advantages across all metrics over non-LLM methods and other confidence-estimation methods on IEMOCAP speech emotion recognition.The comparison uses Qwen2-audio as the base model; Table 3 defines UA as unweighted accuracy and WA as weighted accuracy, but the supplied passages provide no metric values.Source: §4.3 · p. 7, Table 3 · p. 8
• The results indicate that ProbPlug generalizes to multiclass classification and can be applied to multimodal large models.The evaluation trained ProbPlug using Qwen2-audio.Source: §4.3 · pp. 7–8

4.4 Calibration Experiments

• ProbPlug achieves lower average ECE (0.0805) and Brier Score (0.0824) than SAPLMA, while staying closer to the perfect calibration line across most bins.SAPLMA shows clear overconfidence in high-confidence regions; ProbPlug's gains are notable on SMS-SPAM and Amazon-polarity.Source: §4.4 · p. 8

4.5 Ablation Studies

• Both token compression and layer-wise information aggregation improve performance under in-task and cross-task evaluation.The ablations were trained on SMS-SPAM and evaluated on SMS-SPAM and SST2.Source: Table 5 · p. 9, §4.5 · p. 9
• Using 10 tokens slightly improves in-task performance over S=1 but noticeably reduces cross-task generalization; the final token provides better robustness.The authors suggest preceding tokens add task-specific semantic information that can cause source-domain overfitting, whereas the final token captures decision confidence with less task-specific noise.Source: §4.5 · p. 9

4.6 Layer-wise Attention Visualization and Interpretability

• In high-confidence cases, ProbPlug concentrates attention on early lexical layers (L0L5) and final reasoning layers (L30L35), giving intermediate layers little weight.The authors suggest intermediate semantic representations may add task-specific noise; suppressing them keeps attention focused and improves confidence robustness.Source: §4.6 · pp. 9–10

5 Conclusion

• ProbPlug estimates prediction confidence from hidden token representations of a frozen LLM without modifying its architecture.The framework is designed to address uncalibrated confidence in LLM outputs.Source: §5 · p. 10
• Across multiple text and multimodal classification tasks, ProbPlug provides more reliable confidence estimates and consistent downstream classification gains with limited inference overhead.The experiments include standard LLMs such as Qwen3 and multimodal models such as Qwen2-Audio.Source: §5 · p. 10
• ProbPlug shows promising cross-task generalization, reducing the need for extensive retraining when adapting to new tasks.The paper presents this as a potential aid to reliable LLM deployment in high-stakes scenarios requiring trustworthy confidence assessment.Source: §5 · p. 10