Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering

1. Introduction

• Prompt predictions are sensitive to templates, verbalizers, and demonstrations, and prior calibration work lacks a unified analysis distinguishing methods’ merits and downsides.The paper describes the relevant bias as an a-priori propensity to predict certain classes unfairly.Source: §1 · p. 1
• The paper analyzes existing calibration methods through decision boundaries, extending them to challenging evaluation tasks to expose failure cases and method-specific limitations.The analysis motivates a unified, widely applicable solution for diverse bias sources.Source: §1 · p. 2
• BC estimates contextual bias by marginalizing LLM scores over a batch; the labeled-data extension BCL adds one learnable parameter to adapt that estimate.BC is zero-shot, inference-only, and adds negligible computation overhead at the LLM output.Source: §1 · p. 2
• Across more than 10 natural language understanding and image-classification tasks, BCL yields further slight gains with more labeled data, and BC alleviates prompt brittleness across prompt choices.The robustness analysis covers templates, ICL choices and orders, and verbalizers.Source: §1 · p. 2
• BC achieves state-of-the-art zero- and few-shot results with PaLM-2 and CLIP, including the best 1-shot calibration-baseline performance averaged over 13 PaLM 2-S/L classification tasks.The 13-task average is reported for PaLM 2-S and PaLM 2-L.Source: Figure 1 · p. 1, §1 · p. 2

Understanding and Improving ICL

• ICL sensitivity has been linked to pretraining term frequencies and corpora, while other accounts describe implicit Bayesian inference and emergent input-label mapping ability.Prior work also selects templates and examples, and proposes noisy-channel prompting, flipped learning, self-ensembling, k-nearest-neighbor label assignment, and decoder training for few-shot ICL.Source: §2 · p. 2

Bias in ICL and Calibrating LLMs

• Calibration work addresses majority-label, recency, and common-token biases with content-free-token calibration, and domain-label bias with random in-domain tokens.Other studies characterize feature and positional bias or estimate robust decision boundaries with Gaussian-mixture-model prototypical clusters.Source: §2 · pp. 2–3
• Positional bias in multiple-choice tasks has also motivated analysis and permutation-based prior estimation.The cited concurrent studies are Pezeshkpour & Hruschka (2023) and Zheng et al. (2023).Source: §2 · p. 3

3.1. Bias in Prompting and In-Context Learning (ICL)

• ICL predictions can be biased by prompt context, demonstration labels and order, and verbalizer-token frequencies.The background describes majority-label and recency-label biases, as well as bias toward label tokens that occur more frequently in pretraining.Source: §3.1 · p. 3

3.2. Overview of ICL Calibration Methods.

• CC estimates a contextual prior from a content-free input and offsets scores with it; DC averages scores from random in-domain text.CC uses “N/A” as a content-free example, while DC samples random tokens from unlabeled text.Source: §3.2 · pp. 3–4
• PC fits class-specific Gaussian-mixture clusters with EM and assigns labels from the estimated clusters at inference.The fitting process can require repeated EM-GMM iterations to stabilize cluster estimates.Source: §3.2 · p. 4

3.3. Design Principles Behind Calibrations

• CC and DC yield linear decision boundaries, whereas PC’s nonlinear boundary can overfit and become unstable in EM-GMM estimation.The analysis hypothesizes that PC’s instability may worsen on challenging multiclass tasks, while linear boundaries may generalize more robustly.Source: §3.3 · p. 5
• On QNLI, content-free inputs can add bias: “N/A” may suggest entailment, while random DC text may create spurious relations.The authors identify multi-sentence classification as a setting where content-free inputs may be inappropriate prior estimators.Source: Figure 3 · p. 5, §3.3 · p. 6

4. Batch Calibration

• BC estimates each class’s contextual bias by averaging its output scores over the samples in a batch.The method uses this content-based batch estimate rather than relying on content-free tokens.Source: §4 · p. 7
• BC calibrates probabilities by dividing by the contextual prior, equivalently shifting log-probabilities by the estimated mean for each class.Source: §4 · p. 7
• BC is zero-shot, uses unlabeled test samples, and adds negligible computation; its bias estimate can be computed once or updated online.The running estimate can stabilize as more mini-batches arrive.Source: §4 · p. 7
• BCL extends BC for settings with labeled data by introducing γ to control calibration strength and potentially mitigate errors in the unlabeled-data estimate.Source: §4 · p. 7
• BCL selects γ by maximizing an evaluation function, such as accuracy, on labeled data.The method searches sampled strengths within a chosen range.Source: §4 · p. 7
• On QQP and CB, BCL’s labeled-data-learned strength performs close to the test-optimal strength, while γ=1 (BC) is strong but not optimal.The illustration uses a randomly sampled 128-shot labeled set.Source: Figure 5 · p. 7, §4 · p. 8
• BC is also applied to zero-shot CLIP classification by calibrating each class at the model’s output.Source: §4 · p. 8

5.1. Experimental Setup

• BC is evaluated on 13 diverse classification tasks spanning GLUE, SuperGLUE, commonsense reasoning, word disambiguation, sentiment, paraphrasing, and inference, plus three image-classification datasets.Experiments use PaLM 2-S, M, and L and CLIP ViT-B/16; Figure 7 describes zero-shot CLIP ViT-16/B accuracy.Source: §5.1 · p. 9, Figure 7 · p. 9

5.2. Main Experiments

• BC improves over ICL by 8% on PaLM 2-S and 6% on PaLM 2-L, and beats PC by 6% on PaLM 2-S and CC by 3% on average on PaLM 2-L.Across evaluated tasks, BC is described as cheaper and stable, while competing baselines show failures or performance fluctuations.Source: §5.2 · pp. 9–10
• Figure 7 reports zero-shot CLIP ViT-16/B accuracy on image-classification tasks to demonstrate BC with a vision-language model.The supplied figure caption does not provide numerical comparisons.Source: Figure 7 · p. 9
• Across 0–4 shots, BC outperforms the baselines; adding an adjustable strength parameter yields a further 1% average gain from limited labeled data.Performance generally improves with more shots, while the gap between BC and ICL converges on some tasks.Source: §5.2 · p. 10

5.3. Robustness and Ablation Studies

• BC is more robust to ICL-example choices and orders, improves consistently over ICL across 10 prompt templates, and largely recovers performance under unconventional emoji verbalizers.An optimal example order can still improve ICL, and a well-designed template can further enhance BC.Source: §5.3 · pp. 10–11, Figure 8 · p. 10
• BC achieves strong performance with around 10 unlabeled samples, whereas PC requires more than 500 before its performance stabilizes.Figure 9 compares performance across initial unlabeled-set sizes without a running estimate of contextual bias.Source: Figure 9 · p. 10, §5.3 · p. 11

6. Conclusion

• The paper interprets calibration methods through decision boundaries to reveal failure cases, then proposes zero-shot, inference-only BC and adjustable BCL for labeled-data settings.The method is described as easy to implement with negligible computation cost.Source: §6 · p. 11
• BC scales from language-only to vision-language settings, achieving state-of-the-art performance in both modalities and improving robustness to prompt designs.The authors anticipate easier prompt engineering with BC.Source: §6 · p. 11
• The authors identify calibration for generative tasks and extension to more models across modalities as future work.They suspect contextual bias may also occur in short-form generation tasks.Source: §6 · p. 11

Prompt Learning

• Soft prompting appends learnable continuous prompts while freezing model weights, whereas hard prompting learns discrete prompts.The passage characterizes prompt learning as an efficient alternative to full-model fine-tuning.Source: A. Additional Related Work · p. 19
• Soft-prompt methods require gradients and internal model access, while black-box hard-prompt search can use reinforcement learning, gradient estimation, or derivative-free algorithms.These approaches seek more interpretable prompts without accessing model gradients.Source: A. Additional Related Work · p. 19

Test-Time Adaptation

• Test-time batch-normalization methods use test-time statistics to address covariate shift, while BC targets bias in LLM contexts without a source statistic analogous to a vision-backbone BN layer.Prior approaches also mix, interpolate, or average source and target statistics.Source: A. Additional Related Work · p. 19

B. Limitations and Future Work

• BC relies on target statistics from batched inputs to estimate contextual bias, so it still requires a batch of inputs.A running estimate from mini-batches is introduced to stabilize predictions as additional batches arrive.Source: B. Limitations and Future Work · p. 19
• Running estimation from mini-batches is intended to stabilize LLM predictions as more batches arrive.The paper presents it as an approach to the estimation errors that can arise with unlabeled data.Source: B. Limitations and Future Work · p. 19
• Calibration for generative tasks and extension to more models across modalities are left for future work.The authors suggest short-form generation as a possible setting where contextual bias may exist.Source: B. Limitations and Future Work · p. 19

• Table 4 gives details of the datasets used for the evaluation reported in Table 2.Source: Table 4 · p. 20
• |Test| denotes the number of test samples.Source: Table 4 · p. 20
• For some datasets, the validation split is used as the test split because labels are not publicly available.Source: Table 4 · p. 20

• Contextual Calibration averages log-probabilities for three content-free tokens and incurs three additional LLM API costs.The tokens are ‘N/A’, ‘’, and ‘[MASK]’.Source: D. Implementation Details · p. 20
• Domain-Context Calibration repeatedly samples content-free and in-domain inputs from a test-set bag of words, averaging results over 20 repetitions and incurring 20 additional API costs.Source: D. Implementation Details · p. 20
• Prototypical Calibration uses the test set as its unlabeled estimate set, with 100 EM iterations and 100 random initializations, a costly process especially for large test sets.Source: D. Implementation Details · p. 20
• BC computes its correction log-probability term once after all test samples have been seen.Source: D. Implementation Details · p. 20
• BC uses different data setups across experiments, including random per-class k-shot examples, labeled samples for BCL, and sampled examples for robustness and order tests.The labeled BCL data comprise J × 128 randomly selected training samples; the order experiment reorders four sampled ICL examples.Source: D. Implementation Details · p. 20

E. Additional Experiments

• Tables 5 and 6 report accuracy for 0-shot PaLM 2-S and 1-shot PaLM 2-M on language tasks, and zero-shot CLIP ViT-16/B on image tasks.Table 6 additionally includes UCF101, FGVC Aircraft, and DTD.Source: Table 5 · p. 21, Table 6 · p. 21
• Table 7 reports per-example BC performance on 1-shot PaLM 2-S for RTE, while Table 10 lists robustness-experiment verbalizers, including emoji pairs for IDs 8–10.Source: Table 7 · p. 21, Table 10 · p. 26
• Figure 10 visualizes decision boundaries for uncalibrated ICL, existing calibration methods, and BC across all binary classification evaluation tasks.Source: Figure 10 · p. 22
• Figure 11 compares an adaptable batch calibration layer with zero-shot BC as calibration strength varies from uncalibrated ICL at 0 to BC at 1.The figure marks the strength learned from labeled data and the best test strength.Source: Figure 11 · p. 23

F. Prompt Templates

• The prompt-template appendices cover zero-shot and k-shot experiments, as well as RTE robustness templates and verbalizer options including “yes / maybe / no.”Source: Table 8 · p. 24, F. Prompt Templates · pp. 24, 27, Table 9 · p. 25