PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Debiasing NLU Models from Unknown Biases

Prasetya Ajie Utama, Nafise Sadat Moosavi, Iryna Gurevych

arXiv:2009.12303v4cs.CLcs.AIcs.LG

TL;DR

NLU debiasing methods often require researchers to know which biased features to target, limiting their use when biases are unknown. This paper introduces a self-debiasing framework that automatically identifies potentially biased examples for existing methods. Across challenge and cross-dataset evaluations, the framework retains comparable challenge-set gains and suggests better overall robustness, although its annealing mechanism can risk renewed bias reliance.

  • Problem

    Existing debiasing methods assume that the specific biased features are known a-priori, requiring dataset-specific analysis when biases are unknown.

  • Method

    A shallow model automatically identifies potentially biased examples, which existing debiasing methods then down-weight when training the main model.

  • Results

    Self-debiased models obtain equally high challenge-set improvements as known-bias models across three tasks, while self-debiased models outperform the baseline by 1.1pp on average across target datasets.

  • Takeaways & Limitations

    The framework enables existing debiasing methods to address unknown biases without targeting a particular bias and suggests better overall robustness across datasets.

  • Takeaways & Limitations

    The annealing mechanism can cause models to pick up biases again, although setting its parameter close to 1 can retain challenge-set performance while improving in-distribution data.

Abstract

from arXiv · show

NLU models often exploit biases to achieve high dataset-specific performance without properly learning the intended task. Recently proposed debiasing methods are shown to be effective in mitigating this tendency. However, these methods rely on a major assumption that the types of bias should be known a-priori, which limits their application to many NLU tasks and datasets. In this work, we present the first step to bridge this gap by introducing a self-debiasing framework that prevents models from mainly utilizing biases without knowing them in advance. The proposed framework is general and complementary to the existing debiasing methods. We show that it allows these existing methods to retain the improvement on the challenge datasets (i.e., sets of examples designed to expose models' reliance on biases) without specifically targeting certain biases. Furthermore, the evaluation suggests that applying the framework results in improved overall robustness.

1 Introduction

NLU models can exploit superficial biased features, while existing debiasing methods generally require those biases to be known in advance. The paper proposes automatic bias mitigation and reports that self-debiasing preserves challenge-set gains without targeting a specific bias.

  • Prior methods: Existing model-agnostic debiasing methods identify biased examples and down-weight them so models focus on harder examples.Examples include example reweighting, confidence regularization, and model ensembling.
  • Problem: These methods typically assume that the relevant biased features are known a-priori, requiring task-specific analysis that can leave unknown biased patterns undiscovered.The assumed bias may range from word or n-gram cooccurrences to more complex stylistic and lexico-syntactic patterns.
  • Approach: The paper proposes automatically identifying potentially biased examples so existing debiasing methods can operate with little to no prior information about the biases.The proposed strategy is intended to make existing methods applicable without pinpointing a specific bias in advance.
  • Framework: A shallow model is trained to identify potentially biased examples, then used to train the main model through existing debiasing methods.The framework pipelines two models of the same architecture and down-weights examples identified as potentially biased.
  • Evaluation: Across three NLU tasks, self-debiased models obtain equally high challenge-set improvements as models using prior bias knowledge, while cross-dataset evaluation suggests better overall robustness.The framework is designed to retain challenge-set improvements without targeting particular biases.
  • Motivation: Biased features are superficial patterns spuriously associated with target labels, and relying on them can harm fairness or generalization.Examples include protected attributes in hiring systems and surface patterns such as negation words or lexical overlap in NLU.

2 Motivation and Analysis

The analysis studies how large pretrained models rapidly learn synthetic surface shortcuts and uses this behavior to obtain a substitute shallow bias model without precise prior knowledge. Synthetic MNLI experiments show rapid divergence between biased and anti-biased performance, motivating the self-debiasing strategy.

  • Debiasing objective: Debiasing methods seek classifiers that do not mainly use features that are predictive of labels in training data but diverge at test time.These features are represented as b(x), while the desired classifier is f_d.
  • Prior bias identification: Existing methods use a hand-crafted shallow model to estimate bias reliance, but designing its features requires costly dataset-specific analysis.The shallow model directly computes p(y|b(x)) from task-specific bias features.
  • Motivation: The proposed strategy assumes that large pretrained models act as rapid surface learners and therefore quickly overfit to simple bias patterns in small-data settings.This assumption connects early rapid performance gains with reliance on biased surface information.
  • Synthetic bias: Synthetic MNLI datasets inject an artificial feature into 30% of examples, while the remaining 70% remain unchanged.The biased and anti-biased examples differ in whether the injected feature aligns with the target label.
  • Learning dynamics: At an early training point, models reach 100% accuracy on the biased test and almost 0% on the anti-biased test.This trajectory appears across synthetic bias proportions m = {0.9, 0.8, 0.7, 0.6}.
  • Shallow model: The early checkpoint can provide a substitute shallow model, although its predictions are initially low-confidence and become more confident after multiple epochs.Most early predictions fall in the 0.4 probability bin, only slightly above the uniform probability of 0.3.

3 Self-debiasing Framework

The self-debiasing framework automatically identifies potentially biased examples with a shallow model, then applies existing debiasing methods with annealing so the main model can progressively learn from all examples.

  • Framework overview: The framework uses a shallow model to identify potentially biased examples and supplies this information to existing debiasing methods.It is designed to operate without dataset-specific knowledge of bias characteristics.
  • Biased-example identification: The shallow model is trained by fine-tuning a copy of the main model on a small random subset for several epochs, then predicting on unseen training examples.Its correct-label probability indicates how likely each example is to contain bias.
  • Biased-example identification: High shallow-model confidence on the correct label marks an example as potentially biased, whereas overconfident errors identify harder examples for learning.The framework uses the scalar correct-label probability to distinguish these cases.
  • Existing debiasing methods: Example reweighting decreases an instance’s loss contribution as shallow-model confidence in its correct label increases.The weight is computed from one minus the shallow model’s correct-label probability.
  • Existing debiasing methods: Product-of-expert combines the main and shallow models during training, while confidence regularization uses scaled teacher supervision to discourage overconfident predictions on biased examples.Only the main model is optimized in product-of-expert, and only its prediction is used at test time.
  • Annealing mechanism: Annealing gradually lowers the shallow-model bias signal during training, increasing the relative importance of all examples and approaching standard cross-entropy when a=0.The schedule decreases α_t from 1 to a; this mitigates the in-distribution performance loss caused by down-weighting many examples.
  • Annealing mechanism: When a is close to 1, annealing can improve in-distribution performance while retaining reasonably good challenge-set performance, though exposing biased instances risks recovering baseline bias behavior.The authors explicitly identify this risk as a limitation of the mechanism.

4 Experimental Setup

The evaluation tests self-debiasing on three NLU tasks using challenge sets that expose distinct biases, while withholding challenge-set bias information during training and comparing against known-bias settings.

  • Evaluation tasks: The study evaluates natural language inference, fact verification, and paraphrase identification, using challenge test sets strictly for evaluation.The corresponding datasets are MNLI with HANS, FEVER with FeverSymmetric, and QQP with PAWS.
  • Evaluation tasks: MNLI is evaluated on HANS, which tests lexical-overlap bias by including non-entailing sentence pairs with the same words.The passage notes that word overlap in MNLI is associated with entailment.
  • Evaluation tasks: QQP is evaluated on PAWS to test whether paraphrase models rely on lexical-overlap biases.QQP contains 362K question pairs labeled duplicate or non-duplicate.
  • Evaluation tasks: FEVER is evaluated on FeverSymmetric, designed to reduce claim-only biases such as negative phrases associated with refutes.FEVER examples pair claims with evidence and use support, refutes, and not-enough-information labels.
  • Experimental protocol: The framework is applied to BERT and compared with versions of existing debiasing methods trained using prior knowledge of task-specific biases.The comparison covers self-debiasing and known-bias training settings.
  • Experimental protocol: Hyperparameters for the shallow model and annealing are tuned using training data without using information from challenge sets.The selected shallow model is based on its behavior on unseen training examples.

5 Results and Discussion

The self-debiasing framework matches or improves prior-knowledge debiasing on challenge sets, improves cross-dataset robustness, and reveals distinct learning dynamics and stability patterns.

  • Main results: Self-debiased models achieve equally high challenge-set improvements as corresponding known-bias models across three NLU tasks.In one PAWS comparison, self-debias example reweighting improves non-duplicate-subset F1 by 52.3 versus 33.6 for known-bias training.
  • Main results: Annealing mitigates in-distribution performance losses in most cases while preserving challenge-set gains.Without annealing, self-debias PoE is 2.2pp below known-bias training on MNLI development data.
  • Cross-datasets evaluation: Self-debiasing outperforms the baseline on all cross-dataset targets, adding 1.1pp on average with or without annealing.Known-bias training improves SICK and Scitail but reduces accuracy on RTE and GLUE diagnostic by 0.5 and 0.6pp.
  • Analyzing the annealing mechanism: Challenge-test scores decrease as minimum αt decreases, while annealing offers a reasonable in-distribution–challenge trade-off through α=0.6.At α=0, performance falls back to baseline.
  • Impact on learning dynamics: Baseline training converges faster in median loss but produces unusually low-loss examples early, whereas self-debiased training maintains lower loss variability.Baseline examples below its 25th percentile fall below 10^-1 while most losses remain high, consistent with bias-driven optimization.
  • Bias identification stability: Across 10 shallow-model instances, easy subsets consistently outperform hard subsets for both lexical-overlap and hypothesis-only partitions.One rare run fails to converge, likely because biased examples were undersampled; it can be discarded and resampled.

6 Related Work

Related work documents pervasive dataset shortcuts, challenge-set evaluation, and efforts to reduce hidden biases through data construction and learning methods.

  • Dataset biases: Dataset artifacts let NLI models perform above chance using partial inputs or input overlap instead of intended reasoning skills.Similar shortcut phenomena are reported in argumentation mining, reading comprehension, and story cloze completion.
  • Evaluation assumptions: The stability analysis intentionally uses lexical-overlap and hypothesis-only biases to validate whether shallow models consistently capture intended biases.This evaluation uses prior bias knowledge as a validation step rather than as the proposed debiasing requirement.
  • Challenge evaluation: Challenge test sets evaluate reasoning by constructing counterexamples to spurious shortcuts used by models.These sets are designed to expose failures caused by reliance on biased features.
  • Bias mitigation: Adversarial filtering and human-in-the-loop dataset construction aim to reduce biases, but newly constructed datasets may retain hidden patterns.The paper therefore motivates complementing data collection with learning-based debiasing.

7 Conclusion

The paper presents self-debiasing as a general way to mitigate unknown dataset biases without dataset-specific bias discovery, while retaining challenge-set gains and improving robustness.

  • Conclusion: The framework removes the need for thorough dataset-specific analysis to discover bias types for each new dataset.It targets unknown dataset biases rather than a predefined bias characteristic.
  • Conclusion: Existing debiasing methods obtain equally high improvements on several challenge test sets without targeting a specific bias.The framework is adopted as a general wrapper around existing methods.
  • Conclusion: Evaluation suggests better overall robustness than bias-specific counterparts.The paper proposes online identification and discouragement of potentially biased examples as future work.

A Natural Language Inference

The NLI experiments fine-tune BERT for baseline, known-bias, and self-debiasing settings, with a smaller shallow model trained to identify biases.

  • Main model: BERT base is fine-tuned for three epochs at learning rate 5^-5 across baseline, known-bias, and self-debiasing settings.Product-of-expert and confidence-regularization models instead follow prior work with five training epochs.
  • Shallow model: The MNLI shallow model uses 2K examples, three epochs, and learning rate 5^-5.It provides the shallow-model component used by the self-debiasing setup.

B Fact verification

The experiments use BERT-based main models and shallow models trained on small subsets for FEVER and QQP.

  • FEVER main models are fine-tuned BERT base models for 3 epochs at learning rate 2−5.
  • QQP main models are trained for 3 epochs at learning rate 2−5.

D Synthetic MNLI Results

The synthetic-bias experiments report final model accuracy and examine how anti-biased accuracy changes with the proportion of biased examples.

  • Final accuracy is reported for models trained on synthetic bias datasets.
  • Anti-biased accuracy correlates negatively with the proportion of biased examples.

E Detailed HANS Results

The HANS evaluation covers lexical-overlap, subsequence, and constituent phenomena, with results reported on entailment and non-entailment subsets.

  • HANS contains lexical-overlap, subsequence, and constituent subsets involving different inference phenomena.Each subset includes examples of both entailment and non-entailment.
  • MNLI's three-way predictions are mapped to HANS using max pooling between neutral and contradiction labels.
  • Table 4 reports model performance on the HANS challenge test set across lexical, constituency, and subsequence categories.
Loading 2009.12303v4…