Source-linked AI summary
Competency Problems: On Finding and Removing Artifacts in Language Data
Matt Gardner, William Merrill, Jesse Dodge, Matthew E. Peters, Alexis Ross, Sameer Singh, Noah A. Smith
TL;DR
The paper addresses how to distinguish spurious from legitimate feature-label correlations when studying language understanding. It formalizes competency problems, analyzes biased data collection, and reports that even small biases can produce increasingly detectable artifacts as datasets grow.
Problem
Prior work documents dataset artifacts and spurious correlations, but leaves unspecified how to distinguish them from legitimate predictive features, a concern because dataset issues can hinder language understanding.
Source: §1 · p. 1Method
Results
Any amount of modeled sampling bias increases the probability of statistically significant spurious feature correlations as dataset size grows, and the test identifies subtler correlations that models still overweight.
Source: §1 · p. 2Takeaways & Limitations
The authors present their analysis as a starting point for improving NLP data collection and designing model inductive biases that target competency problems.
Source: §1 · p. 2The local-edit guarantees rely on strong independence assumptions that the authors say are not realistically achievable for human-produced language edits.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Much recent work in NLP has documented dataset artifacts, bias, and spurious correlations between input features and output labels. However, how to tell which features have "spurious" instead of legitimate correlations is typically left unspecified. In this work we argue that for complex language understanding tasks, all simple feature correlations are spurious, and we formalize this notion into a class of problems which we call competency problems. For example, the word "amazing" on its own should not give information about a sentiment label independent of the context in which it appears, which could include negation, metaphor, sarcasm, etc. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account, showing that realistic datasets will increasingly deviate from competency problems as dataset size increases. This analysis gives us a simple statistical test for dataset artifacts, which we use to show more subtle biases than were described in prior work, including demonstrating that models are inappropriately affected by these less extreme biases. Our theoretical treatment of this problem also allows us to analyze proposed solutions, such as making local edits to dataset instances, and to give recommendations for future data collection and model design efforts that target competency problems.
1 Introduction
The paper defines competency problems by requiring that no individual feature predict a label, then analyzes how biased data collection creates artifacts and how they can be detected and mitigated.
1 Introduction
- Competency problems require each single feature to provide no information about the label, so prediction depends on feature interactions.The paper argues this is a reasonable description of complex language understanding problems, where a word such as “amazing” can occur in contexts without positive sentiment.Source: §1 · pp. 1–2
- Any amount of sampling bias increases the probability of statistically significant spurious feature correlations as dataset size grows.The analysis models human bias through rejection sampling.Source: §1 · p. 2
- A statistical test for artifacts reveals less extreme but significant correlations that pointwise mutual information can miss, and models still overweight those features.Figure 1 plots word occurrence counts against conditional label probabilities; features above its blue line have detectable correlations under a conservative Bonferroni-corrected test.Source: Figure 1 · p. 1, §1 · p. 2
- The framework analyzes local edits and filtering, deriving conditions for edits to remove artifacts and connecting local-edit analysis to Boolean-function sensitivity.The proofs are intended to help dataset builders monitor whether collected data is artifact-free.Source: §1 · p. 2
- The authors present their analysis as a starting point for future work on data collection methods and model inductive biases for competency problems.The paper also identifies gaps in sensitivity theory relevant to bias in sampled datasets.Source: §1 · p. 2
2 Competency Problems
Competency problems formalize the idea that language understanding should depend on combinations of meaning rather than isolated feature-label correlations.
2 Competency Problems
- A competency problem has a uniform marginal label distribution given any single feature, with the label information residing in feature interactions.In the paper’s Boolean setting, this means p(y|xi) = 0.5 for all i.Source: §2 · p. 2
- The framework treats single-feature signals as extra-linguistic information that must be controlled to test linguistic competence, even when such signals help ordinary prediction.The authors note that the choice of what counts as a simple feature is somewhat arbitrary, though word types are their usual choice.Source: §2 · pp. 2–3
- For sentiment analysis, a word such as “amazing” can be predictively useful through distributional effects yet be a red herring when testing whether a model understands contextual expression.The paper distinguishes exploiting lexical-context conflation for classification from evaluating linguistic competence.Source: §2 · p. 3
3 Biased Sampling
The paper models human bias in data collection as rejection sampling, then analyzes how it creates detectable artifacts and sets up a statistical test for them.
3 Biased Sampling
- Human blind spots, social bias, and priming make competency data difficult to collect; the analysis approximates these effects with rejection sampling from an unbiased distribution.The authors describe rejection sampling as a tractable first-order approximation, not a claim about the actual psychological process.Source: §3 · p. 3
3.1 Emergence of Artifacts Under Rejection Sampling
- Under biased sampling, the expected conditional label probability shifts away from 0.5 as rejection probability rises, while larger samples concentrate around that biased expectation.Consequently, collecting more data from the same biased process does not remove the artifact; it can be exploited at i.i.d. test time but fails under the competency setting.Source: §3.1 · p. 4
3.2 Hypothesis Test
- The proposed one-sided binomial proportion test checks whether the observed conditional label probability differs from 0.5, with its detection threshold depending on sample size and confidence level.The null hypothesis corresponds to no rejection bias, ri = 0.Source: §3.2 · p. 4
4 Empirical Analysis
The analysis uses a statistical test that accounts for feature frequency and label association to identify artifacts, then examines whether models respond to them. It finds numerous dataset deviations and measurable single-token model bias.
4.1 Data Analysis
- SNLI showed many more deviations from the competency assumption than a PMI-based analysis would suggest, including less-extreme correlations that still affect model behavior.Source: §4.1 · p. 5
- Many frequent (verb, preposition) tuples were at or near the significance curve, potentially explaining low contrast consistency through models’ reliance on these artifacts.Source: Figure 2 · p. 5, §4.1 · p. 5
4.2 Model Analysis
- ∆p̃_y exceeded 10% for some classes; bias was more extreme on SNLI than RTE.The comparison uses cohorts of single tokens with high and low z* values.Source: Table 1 · p. 6, §4.2 · p. 6
- Because the experiment used unnatural single-word inputs to avoid nonsensical replacements, measured token effects may not fully reflect model behavior in context.Source: §4.2 · p. 6
5 Mitigating Artifacts with Local Edits
The paper derives conditions for local edits to reduce first-order artifacts, tests edits on BoolQ and IMDb, and connects editing to sensitivity: sparse training neighborhoods can leave sensitivity underspecified, while edited neighborhoods provide lower bounds.
5 Mitigating Artifacts with Local Edits
- Under independence assumptions, local edits yield unbiased data only if the starting data is unbiased, every feature is flipped, or each edited feature changes the label half the time.The authors describe flipping every feature as impractical for language data and recommend roughly half-time label changes, uncorrelated with which features are edited.Source: §5 · pp. 6–7
- These guarantees rely on strong assumptions; the authors recommend treating them as guidelines and monitoring edit sensitivity and artifact statistics.Correlated edit and sensitivity probabilities can break the half-time condition, and the independence assumptions are not realistically achievable for human-produced edits.Source: §5 · p. 7
5.1 Local Edits in Practice
- On BoolQ, almost all edited tokens fell within the confidence region, unlike the original data; on IMDb, original and edited artifact statistics showed no apparent distributional difference.BoolQ's per-token edit sensitivity had median 0.429 (mean 0.348, std 0.274); IMDb's was 1.00 (mean 1, std 0).Source: §5.1 · p. 7, Figure 3 · p. 7
5.2 Local Edits and Boolean Sensitivity
- Sparse sampled neighborhoods can leave true function sensitivity underspecified, so a learner whose complexity tracks sensitivity may favor low-sensitivity alternatives even when the gold function is high-sensitivity.For uniformly sampled boolean inputs, the probability of observing any Hamming neighbors falls rapidly with dimension, and a low-sensitivity function can fit the observed data perfectly.Source: §5.2 · p. 8
- Local edits that fill some neighborhoods make observed neighbors a lower bound on local and global sensitivity, so global sensitivity is no longer underspecified by the dataset.Source: §5.2 · p. 8
- Sensitivity estimates from generated neighbors may be too high for a sampled dataset because the generative model can fill regions missing through sampling bias.The discussion also notes that no precise measure of sampled-dataset sensitivity is established, particularly when generalizing from finite boolean functions to natural-language inputs.Source: §5.2 · p. 8
6 Other Mitigation Techniques
The paper considers increasing annotators and filtering data as artifact-mitigation strategies. Uncorrelated annotator biases can average out, while correlated biases persist; ambiguous-instance filtering is associated with fewer statistical deviations from the competency assumption.
6 Other Mitigation Techniques
- The discussion of other mitigation techniques is explicitly high-level intuition and is not rigorous.Source: §6 · p. 8
- Adding annotators may reduce feature-label bias when their bias vectors are uncorrelated, but correlated biases can persist and prevent the competency assumption from being reflected.The text names societal biases, word-use frequencies, and shared priming from collection instructions as possible sources of correlation.Source: §6 · pp. 8–9
Data filtering
- Removing instances can help with very biased datasets, but bias in a single feature may lead to discarding entire examples and, in extreme cases, effectively removing that feature from the observed space.Source: Data filtering · p. 9
- Ambiguous instances had many fewer deviations from the competency assumption across the hypothesis-test range, not only at high PMI values.The ambiguous set was label-balanced and compared with a downsampled training set of the same size.Source: Data filtering · p. 9
7 Other Related Work
Related work places the competency assumption alongside theoretical analyses of bias and connects ambiguous-instance filtering with fewer statistical artifacts.
7 Other Related Work
- This work differs from other theoretical analyses of bias by introducing the competency assumption; its biased and unbiased distributions instantiate covariate shift.The cited related work includes Shah et al. (2020a), Kaushik et al. (2020), and Quionero-Candela et al. (2009).Source: §7 · p. 9
- Ambiguous instances show fewer deviations from the competency assumption across the full hypothesis-test range; a same-size SNLI sample contains more artifacts.The comparison uses 24% of the data and is consistent with the paper’s result that larger datasets are more likely to contain artifacts.Source: Figure 4 · p. 9, §7 · p. 9
8 Conclusion
The conclusion argues that simple feature-label correlations are spurious for language understanding and that biased data collection makes competency datasets difficult to create. It offers data-collection practices informed by the paper’s theoretical analysis.
8 Conclusion
- Models can learn statistical patterns that let them bypass linguistic competence, motivating the paper’s competency-problem framing.The conclusion characterizes all correlations between simple features and labels as spurious for language understanding.Source: §8 · p. 9
- The authors present their theoretical analysis as a starting point for future data-collection analyses and insights into inductive biases for modeling competency problems.The conclusion also says the analysis can inform future data-collection efforts for such tasks.Source: §8 · p. 10
- For generated text, the authors recommend reducing priming; for existing text, they recommend local edits monitored for sensitivity, potentially using different processes for train and test.Suggested ways to reduce priming include using images as prompts or randomly sampling words to include in generated text.Source: §8 · p. 10
A Ambiguous PP Attachment Extraction
This appendix describes a heuristic procedure for extracting ambiguous-attachment constructions from UD English Web Treebank training data.
A Ambiguous PP Attachment Extraction
- The extraction heuristically targets verb–noun–prepositional-phrase constructions with ambiguous attachment in UD English Web Treebank training data.The section introduces this extraction procedure as a description of the method in Section 4.1.Source: A Ambiguous PP Attachment Extraction · p. 13
- The extraction is framed as heuristic rather than as an exhaustive or fully specified construction-identification method.The section calls the procedure heuristic and directs readers to criteria operating over dependency annotations.Source: A Ambiguous PP Attachment Extraction · p. 13
- The procedure extracts (V, N, PP) constructions from UD English Web Treebank inputs using criteria based on dependency-relation annotations.The supplied passage introduces the criteria but does not enumerate them.Source: A Ambiguous PP Attachment Extraction · p. 13
B Proof of Proposition 1
The proof assumes independence among input features and edit parameters, derives a conditional-label result under pairwise feature independence, and identifies three factorized conditions for unbiasedness.
B Proof of Proposition 1
- Under pairwise independence of input features, Lemma 1 gives p_b(y | ¬x_i) = 1/2.The proof first derives the joint and marginal probabilities before computing this conditional probability.Source: B Proof of Proposition 1 · p. 13
- The proof defines edit sensitivity s_i as the probability that an edit to feature i flips the label.It uses this definition in the proof of Proposition 1.Source: B Proof of Proposition 1 · p. 13
- Proposition 1 assumes independence among input features and their edit parameters across feature pairs.The stated independent quantities are x_i, x_j, e_i, e_j, s_i, and s_j.Source: B Proof of Proposition 1 · p. 13
- The equation factorizes into three conditions for an unbiased p_e(y′ | x′): r_i = 0, e_i = 1, or s_i = 1/2.The text says the implications of these three solutions are discussed in the main text.Source: B Proof of Proposition 1 · p. 14