PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Relation between Sensitivity and Accuracy in In-context Learning

Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, He He

arXiv:2209.07661v3cs.CLcs.AIcs.LG

TL;DR

ICL is oversensitive to prompts, while when and how predictions become sensitive remains unclear. The paper studies perturbations, adjusts sensitivity for label bias, and proposes SENSEL to abstain on sensitive predictions. Across ten classification datasets, SENSEL consistently outperforms confidence- and entropy-based baselines on abstention decisions.

  • Problem

    ICL is known to be oversensitive to prompts, but the conditions and behavior of prediction sensitivity remain unclear, limiting reliability in practice.

    Source: §1 · p. 1

  • Method

    The paper measures sensitivity under instruction and example-order perturbations, mitigates label bias, and uses sensitivity to drive SENSEL abstention decisions.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    SENSEL consistently outperforms MAXPROB and ENTROPY, with gains of up to +4.1 AUC points, while adjusted sensitivity reaches 2.8x raw sensitivity and correlates negatively with accuracy at up to −0.40 Pearson.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    Sensitivity is a useful signal for identifying error-prone ICL predictions and making few-shot selective abstention decisions.

    Source: §1 · p. 1, §4 · p. 5

    The sensitivity-accuracy relationship is correlational rather than causal, and the study mainly focuses on text classification tasks.

    Source: §5 · p. 5

Abstract

from arXiv · show

In-context learning (ICL) suffers from oversensitivity to the prompt, making it unreliable in real-world scenarios. We study the sensitivity of ICL with respect to multiple perturbation types. First, we find that label bias obscures the true sensitivity, and therefore prior work may have significantly underestimated ICL sensitivity. Second, we observe a strong negative correlation between ICL sensitivity and accuracy: predictions sensitive to perturbations are less likely to be correct. Motivated by these findings, we propose \textsc{SenSel}, a few-shot selective prediction method that abstains from sensitive predictions. Experiments on ten classification datasets show that \textsc{SenSel} consistently outperforms two commonly used confidence-based and entropy-based baselines on abstention decisions.

1 Introduction

ICL sensitivity is obscured by label bias, yet after adjustment it is negatively correlated with accuracy. The paper uses this signal for SENSEL, which abstains on sensitive predictions and outperforms probability-based baselines.

  • The paper systematically studies perturbations to task instructions and in-context example order while measuring sensitivity through output changes.This evaluation covers multiple prompt perturbation types rather than a single prompt variation.Source: §1 · p. 1
  • Adjusted ICL sensitivity can reach 2.8x raw sensitivity because label bias makes predictions appear stable across prompts.Language models may favor a particular label regardless of the prompt, masking genuine sensitivity.Source: §1 · p. 1
  • Sensitivity and accuracy show a significant negative correlation, reaching −0.40 Pearson, so sensitive predictions are more likely to be incorrect.The relationship is observed after mitigating label bias and across prompt perturbations.Source: §1 · p. 1, Figure 1 · p. 2
  • SENSEL abstains on examples whose predictions are sensitive to prompt perturbations, using sensitivity as its abstention signal.This avoids requiring the large labeled datasets needed to train a separate abstention model in few-shot settings.Source: §1 · p. 1
  • SENSEL consistently outperforms MAXPROB and ENTROPY on abstention decisions, with gains of up to +4.1 AUC points.The method uses sensitivity rather than output probabilities to identify error-prone examples.Source: §1 · pp. 1–2

2 ICL Sensitivity Study

The study examines how prompt perturbations affect ICL sensitivity and how label bias distorts its measurement. Across ten datasets and two language models, adjusted sensitivity is higher after calibration, negatively related to accuracy, and strongest for human-written instruction changes.

2 ICL Sensitivity Study

  • Prior approaches address ICL sensitivity through fine-tuning, meta-training, or prompt search, whereas this study links sensitivity to label bias and accuracy.The paper positions its analysis as distinct from methods that directly seek more consistent predictions.Source: §2.1 · p. 2

2.1 ICL Sensitivity

  • ICL sensitivity is measured by changes in predicted labels under instruction and example-order perturbations, including human-written, automatic, and ordering variants.The experiments use GPT-J-6B and GPT-NEO-2.7B on ten classification datasets.Source: §2.1 · pp. 2–3, §2.2 · p. 3

2.3 Findings

  • Adjusted sensitivity is 99.0% higher after prototypical calibration, indicating that label bias can substantially underestimate ICL sensitivity.The increase appears across both models and all three perturbation sets.Source: §2.3 · p. 4
  • Human instruction perturbations change predictions 43% of the time on GPT-J-6B and 50% on GPT-NEO-2.7B after label-bias mitigation.This is the strongest sensitivity among the three perturbation sets.Source: §2.3 · p. 4
  • Sensitivity and accuracy show a significant negative correlation across perturbation sets, reaching −0.39 for human instruction perturbations on both models.The study computes Pearson correlations after mitigating label bias.Source: Table 1 · p. 3, §2.3 · p. 4

3 Sensitivity-based Selective Few-shot Prediction

The paper proposes SENSEL, which uses sensitivity to prompt perturbations as a confidence signal for abstaining on likely errors. Across instruction-available and instruction-free settings, SENSEL outperforms probability- and entropy-based baselines, especially on high-sensitivity tasks.

3 Sensitivity-based Selective Few-shot Prediction

  • SENSEL scores confidence as negative sensitivity and abstains when that score falls below a threshold.The threshold controls the trade-off between accuracy and coverage.Source: §3 · p. 4
  • SENSEL consistently outperforms MAXPROB and ENTROPY on selective prediction across the evaluated settings.Performance is measured by the area under the F1-Coverage curve.Source: §3 · p. 5
  • +4.1 AUC points on GPT-J-6B and +5.0 AUC points on GPT-NEO-2.7B are the average margins for SENSEL with human-written instruction perturbations over MAXPROB.This perturbation setting performs best among the three evaluated variants.Source: §3 · p. 5
  • +3.0 AUC points on GPT-J-6B and +1.4 AUC points on GPT-NEO-2.7B are SENSEL-EXORD’s average margins over MAXPROB and ENTROPY without task instructions.The result holds in the NO INST setting.Source: §3 · p. 5
  • MAXPROB performs better on low-sensitivity tasks, while SENSEL performs better on high-sensitivity tasks, with correlations of −0.17 and +0.28 respectively.The analysis compares each method’s abstention AUC against a random-abstention baseline.Source: §3 · p. 5

4 Conclusion

The paper establishes that ICL sensitivity reflects task understanding and is negatively correlated with accuracy after label bias is mitigated. It uses this relationship to motivate selective prediction that abstains on highly sensitive predictions.

  • The study finds that ICL sensitivity is negatively correlated with accuracy when label bias is mitigated.Source: §4 · p. 5
  • The proposed selective prediction method abstains on highly sensitive predictions.Source: §4 · p. 5
  • ICL sensitivity exhibits a useful pattern: it reflects how confidently an LM understands the task.Source: §4 · p. 5
  • The sensitivity–accuracy relationship is correlational rather than causal, and its underlying mechanism remains unclear.Source: §4 · p. 5

5 Limitations

The paper identifies unresolved causal and mechanistic questions about sensitivity and accuracy and notes that its empirical scope mainly covers text classification.

  • The sensitivity–accuracy relationship is correlational, not causal, so whether sensitivity arises because predictions are incorrect remains unresolved.Source: §5 · p. 5
  • The mechanism behind the negative correlation between ICL sensitivity and accuracy remains unclear.Source: §5 · p. 5
  • The study mainly focuses on text classification, leaving tasks such as text generation and question answering for future work.Source: §5 · p. 5

A Datasets

The study evaluates ICL sensitivity and few-shot selective prediction across ten named datasets spanning news, reviews, emotion, question answering, and topic classification.

  • The appendix frames these datasets as the evaluation basis for studying ICL sensitivity and few-shot selective prediction.Source: A Datasets · p. 8
  • The study covers ten datasets, including AG News, Amazon Review Polarity, DBPedia14, Emo2019, CARER, WikiQA, Yahoo Answers Topics, Large Yelp Review, Yelp Reviews Full Star, and Rotten Tomatoes.Source: A Datasets · p. 8

B Sensitivity Study Implementation Details

The implementation uses four-shot prompts averaged over five sampled demonstration sets, label-bias mitigation procedures, and human or automatically generated instruction perturbations.

C.1 ICL Sensitivity Study

After mitigating label bias with contextual calibration, ICL appears more sensitive to prompt perturbations, and sensitivity is negatively correlated with accuracy across datasets.

Confounding Label Bias

  • Label-bias mitigation with CC makes ICL more sensitive across prompt perturbation types than raw sensitivity suggests.The comparison covers instruction paraphrasing, automatic instruction perturbation, and example ordering.Source: §C.1 · p. 8

Sensitivity-Accuracy Correlation

  • Sensitivity and accuracy show a significant negative correlation across datasets for both raw and CC-adjusted measurements.The correlation is reported for the raw and CC conditions in Table 3.Source: §C.1 · p. 8

C.2 Sensitivity-Based Selective Few-shot Prediction

SENSEL-based selective prediction consistently outperforms probability-based baselines across perturbation settings, models, and evaluation thresholds.

C.2 Sensitivity-Based Selective Few-shot Prediction

  • SENSEL consistently outperforms MAXPROB and ENTROPY across perturbation variants under both INST and NO INST settings.This result holds with contextual calibration on both GPT-J-6B and GPT-NEO-2.7B; human-written instruction perturbations perform best among the variants.Source: §C.2 · p. 8, Figure 5 · p. 9, Table 4 · p. 12
  • SENSEL-INSTH achieves higher coverage than MAXPROB at different F1 thresholds for GPT-J-6B.The comparison is shown using Coverage-F1 curves with confidence bands.Source: §C.2 · p. 8, Figure 6 · p. 9
  • Adjusted sensitivity exceeds raw sensitivity across human instruction, automatic instruction, and example-ordering perturbations after label-bias mitigation.Table 2 reports standard deviations across five randomly sampled few-shot example sets.Source: Table 2 · p. 10
  • The reported sensitivity-accuracy relationship is strongly negative across all three perturbation sets.Table 3 reports Pearson correlations and standard deviations across five randomly sampled few-shot example sets.Source: Table 3 · p. 11
Loading 2209.07661v3…