Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

1 Introduction

• Different templates can make reported differences between ICL methods reflect format variation rather than the methods themselves.Prior studies often used a specific template without explaining its selection, or compared results evaluated on different template sets.Source: §1 · p. 1
• The best template depends on the evaluation setting and does not reliably transfer across models, demonstration sets, or prediction methods.Even slight changes can make a strong template perform poorly, complicating template tuning.Source: §1 · p. 2
• Template Ensembles average predictions across several prompt formats to improve average performance and reduce sensitivity to template choice.Source: §1 · p. 2
• Across 21 models and 4 datasets, selecting a suitable template alone can produce gains similar to proposed ICL improvements.Source: §1 · p. 2

2.1 In-Context Learning

• In-context learning classifies a test input by choosing the label that gives the model the highest-probability sequence, with demonstrations formatted as a natural-language input.Templates use input and output verbalizers, an intra-separator, and an inter-separator.Source: §2.1 · p. 2

2.2 In-Context Learning Analysis

• Prompt components can cause unpredictable performance changes, and prior template-robustness work studied minimal or task-specific formats and optimized templates.The paper also uses verbalizers from Gao et al. (2021) alongside minimal and universal templates.Source: §2.2 · pp. 2–3
• Example-selection methods include ITM, which learns latent task-concept variables, and Z-ICL, which retrieves similar unlabeled examples and assigns them random labels.Both had been evaluated on different single templates, leaving method gains difficult to distinguish from template effects.Source: §2.2 · p. 3
• Prediction approaches include CALIBRATION, which corrects predictions using a placeholder input, and CHANNEL, which maximizes P(x|y) rather than P(y|x) (Min et al., 2022b).Prior comparisons used separate template sets, motivating evaluation across broader template and model ranges.Source: §2.2 · p. 3

3.1 Models and Data

• The evaluation uses models from frequently studied families such as OPT and BLOOM, as well as newer models such as Llama 2 and Falcon, across four classification datasets.The datasets are SST-2, DBPedia, AGNews, and TREC.Source: §3.1 · p. 4
• Combining the template-component options yields 216 prompt formats for SST-2 and 168 for each of DBPedia, AGNews, and TREC.Evaluating all models on 10 random templates in one setup takes 17–48 hours on one NVIDIA A100-80GB GPU.Source: §3.1 · p. 4

3.2 Methods

• For each setting, results are aggregated over 3 random seeds for example selection, with 10 random templates per seed, and reported as mean and standard deviation of accuracy.Source: §3.2 · p. 4
• The study compares four-shot ITM and Z-ICL example selection with random selection, and evaluates DIRECT against CHANNEL and CALIBRATION prediction methods.ITM examples selected using GPT2 Large checkpoints are reused across models because training the concept model separately was considered infeasible.Source: §3.2 · pp. 4–5

4.1 Baseline Results

• Even the largest models show score standard deviations up to 35% of their mean values across prompt formats.The evaluation also finds that model-size and demonstration-count trends persist despite this variability.Source: §4.1 · p. 5

4.2 Prediction Methods

• CHANNEL and CALIBRATION generally improve on DIRECT, but their scores substantially overlap DIRECT for some models and datasets.CALIBRATION achieves the highest mean accuracy more often, while being more template-sensitive than CHANNEL.Source: §4.2 · p. 5

4.3 Example Selection Methods

• The evaluation compares ITM and Z-ICL with RANDOM using DIRECT prediction and four demonstrations.The authors note that their setup differs from the original studies and that their ten random templates may omit the templates those studies used.Source: §4.3 · pp. 5–6
• Advanced example-selection methods often perform worse than RANDOM when template sensitivity is considered.ITM often raises average performance but has high across-template variation; Z-ICL is more consistent but performs worse.Source: §4.3 · p. 5

5.1 Setup

• The top-10 templates average 90% of the best-template score, and shared membership in both setups' top tens defines successful transfer.The study compares top-template sets using Intersection-over-Union (IoU).Source: §5.1 · p. 6
• Spearman rank correlation can rise when low-performing templates rank similarly, even while efficient-template transfer remains low.The authors provide this metric's results in Appendix I.Source: §5.1 · p. 6

5.2 Transfer Between Models

• IoU exceeds 0.5 for only a few model pairs, indicating generally low transfer, including between models in the same family.The comparison concerns top-10 templates for base models with two random demonstrations and DIRECT prediction on DBPedia.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 Transfer Between Prediction Methods

• IoU between prediction methods is low, so authors must tune templates for each method they combine with a new ICL modification.The reported Table 4 comparison averages results across 19 base models in the RANDOM 2-shot setup.Source: §5.3 · p. 7, Table 4 · p. 7

5.4 Transfer Between Demonstration Selection Methods

• Adding demonstrations can significantly change the ranking of the best templates, even when they come from the same selection method.The authors conclude that example-selection methods should be evaluated across a range of templates to avoid misinterpreting results.Source: §5.4 · p. 7, Figure 5 · p. 8

5.5 Discussion

• Using a shared format across models can underestimate a method except on the model for which that format was tuned.The authors say fair comparisons require describing and applying the format-selection procedure in every evaluated setting; alternatively, evaluation can aggregate over templates or use each setting's best template, with added variance or computational cost, respectively.Source: §5.5 · pp. 7–8

• Template Ensembles average label probabilities across N templates and predict the label with the highest average probability.This test-time approach uses pretrained-model predictions without additional training, unlike a cited approach that trains a boosting algorithm on model outputs.Source: §6 · p. 8
• Ensembles of size 4 or 5 achieve the best accuracy for most models and prediction methods; further expansion is less effective.Smaller ensembles can be unstable, and using N templates requires N times as many model runs as single-format evaluation.Source: §6 · p. 8
• Ensembles increase accuracy for all evaluated models and prediction methods, while significantly reducing template-choice variance in most setups.The authors conclude that this preserves accuracy gains from ICL modifications while mitigating template sensitivity.Source: §6 · p. 9

• Template selection can affect prediction accuracy as much as the choice of demonstrations or prompting methods.Source: §7 · p. 9
• Template Ensembles do not yet address language models' general sensitivity to minor prompt variations.Source: §7 · p. 9

Limitations

• Evaluation covers four classification datasets, compares only two example-selection methods against random, and leaves label mapping, including random labels, unexplored.Source: Limitations · p. 9
• The study examines template-selection effects and transfer, not why templates affect performance; explaining those effects remains future work.Source: Limitations · p. 9

A Templates from Prior Work

• Templates used by the evaluated methods’ original works had no intersection, complicating comparisons such as CHANNEL outperforming CALIBRATION in Min et al. (2022b).Source: A Templates from Prior Work · p. 14

B Model Selection

• The final model pool retained models with at least eight 2-shot wins over 0-shot, selecting for models that demonstrated an ICL performance boost.Source: B Model Selection · p. 14

C Full Baseline Results

• Template sensitivity appeared across all 19 models regardless of size and was not efficiently mitigated by increasing the number of demonstrations.Source: C Full Baseline Results · p. 14

D Template Parts Analysis

• Template components interact: a verbalizer that ranks poorly alone can appear in a best-performing combination, and component rankings differ across models.Source: D Template Parts Analysis · p. 14

E Prediction Methods

• Accounting for template-driven accuracy spread makes the advantages of advanced prediction strategies over DIRECT less apparent, and neither consistently reduces sensitivity across models and datasets.Source: E Prediction Methods · p. 14

F Reproduction of Results for Advanced Selection Methods

• For ITM and Z-ICL, changing templates decreased mean performance for multiple models and increased seed variance; tokenization differences also separated reproduced results from published scores.Source: F Reproduction of Results for Advanced… · pp. 14, 18, Table 12 · p. 18, Table 13 · p. 18

G Example Selection Methods

• Advanced example-selection techniques often performed comparably to random choice across templates, while finding each setup’s best format was costly and difficult.Source: G Example Selection Methods · p. 20

H Accuracy as a Function of Template Rank

• Within 0.9 of the best-template score, the tenth-best template on SST-2 and AGNews counted as successful transfer; DBPedia and TREC decayed faster.Source: H Accuracy as a Function of Template… · p. 20

I Transfer Evaluation with Spearman Rank Correlation

• Spearman rank correlation gave higher template-transfer correlations than IoU over the top ten formats, but transfer remained far from perfect on datasets including SST-2 and TREC.Source: I Transfer Evaluation with Spearman… · p. 20

J IoU Transfer For All Datasets

• Across all evaluated datasets, overlap among models’ ten best-performing templates was remarkably low.Source: J IoU Transfer For All Datasets · p. 20

K Additional Results For Template Ensembles

• For most evaluated setups, an ensemble of five templates performed better than a single template.Source: K Additional Results For Template… · p. 20

L Evaluation of Instruct Models

• Increasing demonstrations generally improved instruction-tuned models’ performance but did not significantly reduce score variance.Source: L Evaluation of Instruct Models · p. 20, Table 17 · p. 23
• After accounting for template robustness, DIRECT performed on par with CHANNEL and CALIBRATION, sometimes with lower score variance.Source: L Evaluation of Instruct Models · p. 20, Table 18 · p. 23
• Z-ICL demonstrations were consistently worse than those from two other methods, while ITM was slightly more robust to template choice for instruction-tuned models.Source: L Evaluation of Instruct Models · pp. 20–21, Table 19 · p. 24

L.1 Ensemble Results

• Template Ensembles improved mean classification accuracy and significantly reduced variance across templates in every tested instruction-tuned configuration.Source: §L.1 · p. 21, Table 20 · p. 24