PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Label Biases for In-context Learning

Yu Fei, Yifan Hou, Zeming Chen, Antoine Bosselut

arXiv:2305.19148v3cs.CLcs.AIcs.LG

TL;DR

ICL is sensitive to example choice and order, but label biases have received limited systematic categorization and mitigation. The paper defines three bias types and proposes domain-context calibration using random in-domain words. Across datasets, calibration improves performance, especially on tasks with large domain-label bias.

  • Problem

    ICL is sensitive to example choice and order, yet biases in these design settings need systematic categorization, detection, and comprehensive mitigation.

    Source: §1 · p. 1

  • Method

    Domain-context calibration estimates label biases from random in-domain words, optionally with real examples prepended, and uses the estimate to calibrate model output probabilities.

    Source: §1 · p. 2

  • Results

    Across datasets, DC improves Macro-F1 by 20% for GPT-J and 18% for GPT-3; on the highest-bias tasks, gains reach 37% and 35%, respectively, over other baselines.

    Source: §6 · p. 5

  • Takeaways & Limitations

    Dataset bias levels vary, so ICL analyses should select datasets with diverse bias types and report stratified results beyond single aggregate scores.

    Source: §9 · p. 9

    The analysis and calibration cover English classification, not multilingual or open-ended tasks.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Various design settings for in-context learning (ICL), such as the choice and order of the in-context examples, can bias a model toward a particular prediction without being reflective of an understanding of the task. While many studies discuss these design choices, there have been few systematic investigations into categorizing them and mitigating their impact. In this work, we define a typology for three types of label biases in ICL for text classification: vanilla-label bias, context-label bias, and domain-label bias (which we conceptualize and detect for the first time). Our analysis demonstrates that prior label bias calibration methods fall short of addressing all three types of biases. Specifically, domain-label bias restricts LLMs to random-level performance on many tasks regardless of the choice of in-context examples. To mitigate the effect of these biases, we propose a simple bias calibration method that estimates a language model's label bias using random in-domain words from the task corpus. After controlling for this estimated bias when making predictions, our novel domain-context calibration significantly improves the ICL performance of GPT-J and GPT-3 on a wide range of tasks. The gain is substantial on tasks with large domain-label bias (up to 37% in Macro-F1). Furthermore, our results generalize to models with different scales, pretraining methods, and manually-designed task instructions, showing the prevalence of label biases in ICL.

1 Introduction

The introduction frames ICL’s sensitivity to prompt design as a need for systematic bias categorization and mitigation. It introduces three label-bias types, a corpus-word-based calibration method, and reports broad improvements from that method.

1 Introduction

  • ICL’s sensitivity to example choice and order motivates systematic work to categorize, detect, and comprehensively mitigate biases in prompt design.Prior work found that models can be biased toward predictions such as the last in-context example’s label.Source: §1 · p. 1
  • The paper distinguishes vanilla-label bias from label names, context-label bias from prompts, and domain-label bias from the task corpus.Context-label bias includes preferences for majority and last-example labels.Source: §1 · pp. 1–2
  • On many tasks with large domain-label bias, LLMs perform no better than random regardless of the in-context examples, and Contextual Calibration does not remove that effect.Figure 1 contrasts the hate-label skew from random in-domain words with no such bias on SST-2.Source: Figure 1 · p. 1, §1 · p. 2
  • DC estimates label biases using random in-domain words, optionally with real examples prepended, then uses that estimate to calibrate output probabilities.Random words limit semantic meaning while in-domain words account for task-corpus effects.Source: §1 · p. 2
  • Across 24 classification datasets, DC improves average ICL performance by 20% for GPT-J and 18% for GPT-3; high-bias tasks gain up to 37% in Macro-F1.DC also benefits models with different scales, instruction-tuning, task instructions, and zero-shot RoBERTa prompting.Source: §1 · p. 2
  • Severe label bias can obscure models’ actual behavior, so ICL studies need diverse dataset selection and fine-grained analysis.The authors call for accounting for dataset confounders when selecting tasks.Source: §1 · p. 2

2 Categorizing Label Biases in ICL

The section defines vanilla-, context-, and domain-label bias in classification ICL, then shows that GPT-J rarely beats random baselines on datasets with large domain-label bias and that contextual calibration does not address it.

2.1 A Typology of Label Biases

  • The typology separates label-name preference (vanilla-label bias), prompt effects (context-label bias), and task-corpus effects (domain-label bias).Source: §2.1 · p. 3

3 Domain Label Bias

  • 0.95 (hate) versus 0.05 (neutral) were GPT-J’s label priors on random in-domain words in the hate-speech task.Source: §3 · p. 3
  • Domain-label bias is quantified as the distance between label priors under random English and random in-domain words.Source: §3 · p. 3
  • On large-domain-bias datasets, GPT-J rarely beats random baselines, and contextual calibration fails to handle domain-label bias.Source: §3 · p. 3
  • Domain-label bias depends on the model and task formulation, particularly the choice of label names.Source: §3 · p. 3

4 Domain-context Calibration

Domain-context Calibration (DC) estimates label bias from random words sampled from the unlabeled task data, then uses that estimate to recalibrate predictions.

4 Domain-context Calibration

  • DC forms a bag of words from unlabeled task texts and samples random in-domain texts with length equal to the dataset’s average text length.The resulting text reflects the dataset’s word distribution while remaining nearly content-free because it is ungrammatical and may include words from all classes.Source: §4 · p. 4
  • It repeats the sampling process M times and averages the model’s label probabilities for those random texts, optionally prepended with context examples.These probabilities estimate the task’s overall label bias.Source: §4 · p. 4
  • For each example, DC recalibrates the original probability assigned to each label using the estimated bias, then predicts from the resulting estimate.The original probability is P(y|xi, C), the model’s probability for label y on example xi with context C.Source: §4 · p. 4

5 Experimental Setup

Across 24 classification datasets, domain-context calibration improves ICL performance, with larger gains on tasks with greater domain-label bias. The experiments also examine its robustness across model scales, example counts, task instructions, calibration choices, and zero-shot prompting.

5 Experimental Setup

  • Experiments cover 24 text-classification datasets, GPT-J (6B) and GPT-3 (175B), usually with 8 randomly sampled examples and five random seeds.The study compares Macro-F1 against random, uncalibrated, and contextual-calibration baselines.Source: §5 · p. 4

6 Experimental Results

  • DC improves average Macro-F1 by 20% for GPT-J and 18% for GPT-3, with gains up to 37% and 35%, respectively, over other baselines on the highest-bias tasks.On those tasks, DC is the only method reported to significantly outperform the random baseline.Source: Figure 6 · p. 5, §6 · p. 5

6.1 Generalizability

  • Across GPT-3 sizes from 350M to 175B, scaling alone does not ensure better performance on high-bias tasks, while DC improves performance and reduces variation across example choices.The scale comparison covers three datasets with large domain-label bias.Source: Figure 7 · p. 5, §6.1 · p. 5
  • Adding examples does not resolve domain-label bias, and DC performs best across tested settings; task instructions particularly improve GPT-3 with DC, while DC also improves Instruct-GPT3 where CC hurts.Table 1 covers 0, 8, and 16 examples; Table 2 reports average Macro-F1 on three tweet datasets.Source: §6.1 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6

6.2 Analysis

  • On sentiment tasks, calibrating with “N/A” biases GPT-J toward positive predictions, while longer calibration sequences improve performance and sequence length preferences track evaluation-text length.Longer texts prefer longer random calibration texts, and shorter texts prefer shorter ones.Source: §6.2 · p. 6, Figure 10 · p. 7, Figure 11 · p. 7
  • More random in-domain words better capture dataset word distributions and balance TweetEval-hate predictions; random English words do not effectively remove the hate-label bias.The reported prediction bias is gradually mitigated as more in-domain words are used.Source: §6.2 · pp. 6–7, Figure 9 · p. 7

6.3 Zero-shot Prompting

  • DC improves RoBERTa-large zero-shot prompting performance by 26% across the 24 datasets, with label-bias mitigation accompanying the gain.This experiment evaluates zero-shot prompting with a smaller model.Source: §6.3 · p. 7

7 Discussion

The discussion examines whether label-name choice can reduce domain-label bias and warns that dataset bias levels complicate aggregate ICL analysis. It also raises adversarial word-level features as a possible source of domain-label bias.

Label name selection as label bias mitigation

  • After DC, the task-indicative labels (neutral, hate) yield the best performance, while (favor, against) underperform the meaningless labels (X, Y).The comparison is on TweetEval-hate; less task-relevant labels mitigate domain-label bias but limit performance.Source: Label name selection as label bias… · p. 8, Figure 12 · p. 8
  • Different domain-label bias levels can make ICL performance vary across datasets, so macro-averaging may obscure task-level differences.The authors recommend selecting datasets to cover varying bias levels and reporting fine-grained results for individual datasets.Source: Label name selection as label bias… · p. 8
  • Adversarial word-level features may contribute to large domain-label bias because LLMs use word associations learned during pretraining for ICL.The discussion connects this possibility to hard examples used to assess robustness in applications such as hate-speech detection.Source: Label name selection as label bias… · p. 8

8 Related Work

Related work studies several sources of ICL sensitivity and label bias, while this paper organizes label biases into a broader typology. The authors also note that testing the largest models remains an open question because they lacked access to them.

Related work

  • Prior work links ICL performance to example order (Lu et al., 2021), pretraining frequencies (Razeghi et al., 2022), example similarity (Liu et al., 2021), pairing format (Min et al., 2022), and prompt structure (Sorensen et al., 2022).The paper positions its study as complementary, focusing on label bias and word-level associations learned during pretraining.Source: §8 · p. 8
  • Zhao et al. (2021) address majority, recency, and common-token biases, whereas this paper proposes a typology and calibration intended to handle all its identified label-bias types.Holtzman et al. (2021) instead study competition among answer surface forms in zero-shot prediction.Source: §8 · p. 8
  • The authors could not test PaLM or Flan-PaLM, leaving their vulnerability to domain-label bias and response to DC untested.They suggest similar behavior between PaLM and Instruct-GPT, and Instruct-GPT's observed domain-label bias, as indications that more capable models may also be susceptible.Source: §8 · pp. 8–9

9 Conclusion

The conclusion presents a three-type label-bias typology and domain-context calibration, emphasizing both improved ICL performance and the need for dataset-aware reporting.

Conclusion

  • The paper categorizes vanilla-label and context-label bias and identifies domain-label bias as a new type that significantly influences ICL performance.Source: §9 · p. 9
  • Domain-context calibration significantly improves ICL performance across a wide range of tasks, particularly on datasets with large domain-label bias.Source: §9 · p. 9
  • Because domain-label bias varies across datasets, ICL analyses should select datasets spanning diverse bias types and report stratified results beyond a single aggregate score.Source: §9 · p. 9

Limitations

The limitations concern the study's language, task, model, and dataset coverage. The ethics statement describes the approach as inference-only and reports no anticipated significant ethical issues from the work.

Limitations

  • The analysis and calibration focus on English classification tasks and word-level domain bias, leaving multilingual tasks, open-ended generation, and other domain-associated biases for future work.GPT-3 experiments cover only a subset of the 24 reported datasets because of budget limitations.Source: Limitations · p. 9
  • Experiments cover GPT models and RoBERTa as the small-scale model, and parts of the GPT-3 evaluation require access to the OpenAI API.Source: Limitations · p. 9

Ethics Statement

  • The tuning-free approach uses off-the-shelf models for inference without pretraining, retraining, or fine-tuning, and the authors anticipate no significant ethical issues from the publicly available text-classification datasets used.They note that the bias analysis may motivate future work in higher-impact areas such as healthcare or legal scenarios.Source: Ethics Statement · p. 9

A Domain-label Bias of All Datasets

Domain-label bias varies across datasets and task types, while estimates from different language models are highly correlated. The appendix also reports that DC’s gain over CC is greater on tasks with more severe domain-label bias.

A Domain-label Bias of All Datasets

  • Detection tasks have the largest domain-label bias, NLI tasks the least, and sentiment and topic bias varies by dataset domain.Movie-review sentiment datasets such as SST-2 have relatively small bias, while financial-statement and poem sentiment datasets have larger bias.Source: A Domain-label Bias of All Datasets · p. 11

B Correlation of Domain-label Bias Estimated with Different LLMs

C Full Dataset Information

  • The evaluation uses 24 datasets spanning sentiment and topic classification, NLI, and detection.Detection datasets were added because those tasks tend to show the largest domain-label bias; evaluation uses test sets when available and development sets otherwise.Source: C Full Dataset Information · pp. 11–12

D Full Few-shot Results

E Templates and Task Instructions

The appendix specifies dataset templates and label-name conventions, then examines how random-text sampling choices affect domain-context calibration’s prior estimates.

E Templates and Task Instructions

  • Templates are provided for all 24 datasets, adapted mainly from prior work and unified for similar tasks after removing task instructions.The task instructions used in the main experiments are listed separately.Source: E Templates and Task Instructions · p. 12
  • Label names use one lower-case word to avoid subword tokenization, except for tasks whose labels are True or False.The appendix also presents the task instructions used in Table 2.Source: E Templates and Task Instructions · p. 12

F Sampling Analysis

  • Sampling analysis tests prior-estimation text counts and unlabeled-corpus sizes on SST-2, AG News, TweetEval-hate, and TweetEval-irony.The selected datasets include two with small and two with large domain-label bias.Source: F Sampling Analysis · p. 12
  • 20 random texts provide a good prior estimate and balance computational efficiency with stability.Figure 15 reports five random-seed runs for each sample count; its caption says 20 texts already yield a stable estimate.Source: F Sampling Analysis · p. 12, Figure 15 · p. 12
  • 50 unlabeled dataset texts provide a good estimate for domain-context calibration.Figure 16 compares corpus sizes and reports five random-seed runs per size; its caption says 50 texts already yield a stable estimate.Source: F Sampling Analysis · p. 12, Figure 16 · p. 12

G Zero-shot Prompting Experiment

The zero-shot prompting experiments adapt existing templates and report RoBERTa-large results on individual datasets. The supplied captions identify the datasets, templates, instructions, and result tables, but do not provide result values.

G Zero-shot Prompting Experiment

Related papers

Loading 2305.19148v3…