PaperlayerView exact revision on arXiv

Source-linked AI summary

UNMASK: Discovering and Causally Verifying Spurious Shortcuts in Text Classifiers

Chidaksh Ravuru, Shashank Srivastava

arXiv:2608.09209v1cs.CLcs.LG

TL;DR

Text classifiers can exploit spurious surface patterns, while existing discovery methods depend on human-defined features or restrictive assumptions. UNMASK automatically discovers and causally verifies such shortcuts, enabling annotation-free mitigation that improves robustness across NLI and toxicity benchmarks and generalizes to reward-model data.

  • Problem

    Existing methods require human intervention or restrictive assumptions, leaving it unclear how to identify unknown spurious features and verify model-level exploitation.

  • Method

    UNMASK uses executable boolean patterns, statistical replication, counterfactual interventions, and annotation-free group definitions to discover, verify, and mitigate spurious correlations.

  • Results

    UNMASK recovers architecture-dependent shortcuts, improves BERT HANS accuracy by up to 12.58 pp, matches hand-labeled DFR on CivilComments, and generalizes to RewardBench2.

  • Takeaways & Limitations

    Programmatic causal groups can replace manual annotations for debiasing while distinguishing dataset correlations from model-exploited shortcuts across tasks and architectures.

  • Takeaways & Limitations

    UNMASK cannot discover latent, semantic, topical, stylistic, or distributional shortcuts that cannot be represented as deterministic predicates over input text.

Abstract

from arXiv · show

Neural language models trained on large crowdsourced corpora frequently exploit spurious surface patterns tied to target labels without true linguistic or causal relevance, boosting benchmark performance while failing on adversarial or out-of-distribution inputs. Existing approaches either require manual specification of the feature vocabulary or automate discovery only partially, leaving the gap between dataset-level correlation and model-level exploitation unaddressed. We present U N M ASK, a fully automated pipeline that discovers, causally verifies, and mitigates spurious correlations in text classifiers without additional human annotation. Given unlabeled training examples, U N M ASK generates candidate surface patterns as executable boolean expressions, filters them through a statistical validation protocol with independent replication, and establishes causal model dependence via verified counterfactual interventions. Causally confirmed features then serve as annotation-free group definitions for Deep Feature Reweighting, eliminating the group labels that standard DFR requires. Applied to BERT and RoBERTa trained on MNLI, our pipeline independently rediscovers established lexical-overlap and negation biases, verifying 9 of 10 features on BERT and 6 on RoBERTa, and improving HANS accuracy by up to 12.58 pp. On CivilComments-WILDS, programmatic groups match the 70.1% worst- group accuracy of hand-labeled DFR (Kirichenko et al., 2023) without demographic annotation. We further demonstrate that the discovery and validation stages generalize to reward model preference data, surfacing interpretable spurious correlations in RewardBench2.

1 Introduction

UNMASK is introduced as a fully automated pipeline that discovers, causally verifies, and mitigates spurious surface correlations in text classifiers without additional human annotation. It uses executable boolean predicates, statistical replication, counterfactual interventions, and the same verified features as group labels for annotation-free DFR.

  • Motivation and problem: UNMASK addresses the need to identify spurious surface features without knowing them in advance and verify whether trained models actually exploit them.Existing approaches require human intervention, restrictive assumptions, a bias-free reference corpus, or opaque model-error proxies.
  • Discovery: An LLM generates candidate shortcuts as deterministic executable boolean functions over input text, enabling corpus-level hypothesis testing rather than descriptive feature naming.Candidates are deduplicated, repaired, and merged when they have near-identical realised coverage.
  • Statistical validation: Statistical validation retains features whose label associations survive Fisher’s exact test with Benjamini–Hochberg control under a two-phase independent replication protocol.This filtering stage is designed to distinguish reproducible associations from candidate patterns that do not validate statistically.
  • Causal verification: Minimal counterfactual edits remove each surviving surface pattern while preserving the semantic label, establishing causal model dependence through paired prediction shifts.The protocol separates dataset-level correlation from model-level exploitation.
  • Mitigation and evaluation: Verified boolean expressions become annotation-free group labels for Deep Feature Reweighting, extending mitigation without requiring additional human group annotations.UNMASK is evaluated on MNLI using BERT and RoBERTa and on CivilComments-WILDS.

2 Related Work

Prior work documents widespread spurious surface patterns in NLP and develops supervised, attribution-based, and programmatic mitigation approaches, but these often require feature identification or rely on restrictive assumptions. UNMASK instead generates deterministic boolean expressions and causally verifies model reliance before debiasing.

  • Spurious correlations: Crowdsourced NLP datasets contain label-correlated surface regularities, including lexical-overlap and negation heuristics in NLI, that do not reflect intended tasks.Related artifacts also occur in fact verification and multimodal tasks, while annotation biases can intensify as datasets scale.
  • Limitations of prior work: Earlier mitigation methods use manually labeled features or LLM-generated debiased data, while automated cross-dataset attribution discovery assumes the pattern is absent from a reference corpus.That assumption is often violated when biases arise from shared annotation processes.
  • UNMASK: UNMASK produces deterministic boolean expressions and verifies model-level causal reliance before applying debiasing.This distinguishes its discovery pipeline from methods that identify correlations without establishing causal model dependence.
  • UNMASK: Programmatic weak supervision also uses boolean text predicates, but its hand-written labeling functions generate labels, whereas UNMASK uses LLM-generated expressions to target spurious features and enable causal verification.The causal verification stage is absent from the weak-supervision paradigm.
  • Contrastive data generation: Contrastive NLI example generation progressed from brittle rule-based templates and manual annotation toward LLM-guided debiasing and relation-aware counterfactual generation.These approaches support targeted evaluation or data generation but do not describe UNMASK’s automated causal feature-discovery pipeline.
  • Mitigation strategies: Mitigation strategies include Product-of-Experts, bias-amplified models, Group DRO, and annotation-free variants that infer groups without explicit bias specification.These methods address identified biases through auxiliary-model weighting, failure upweighting, or worst-group optimization.

3 Methodology

UNMASK separates data-level artifact discovery from model-level reliance analysis, using unlabeled examples to generate executable candidate patterns, statistically validate them, and causally test model dependence through counterfactual interventions. Validated, causally exploited features can then support annotation-free mitigation.

  • Problem formulation: UNMASK defines spurious correlations as label–attribute pairs whose conditional label distribution differs from the marginal, then distinguishes all artifacts from the subset exploited by the trained classifier.The methodology therefore targets both artifact discovery and model-level reliance.
  • Candidate discovery: An LLM generates unlabeled-data surface patterns as executable boolean expressions, returning descriptions, category tags, justifications, and deterministic TRUE/FALSE evaluators.Withholding labels prevents label-specific information from guiding candidate generation.
  • Candidate refinement: Candidates undergo expression execution and repair, followed by coverage-aware deduplication that combines description similarity with Jaccard overlap of firing examples.The second pass retains the highest-coverage candidate within each cluster and avoids merging patterns with largely disjoint coverage.
  • Statistical validation: Spurious Feature Validation uses label-stratified discovery and validation splits, Fisher’s exact tests with Benjamini-Hochberg false-discovery control, held-out replication, and coverage and precision filters.Surviving feature–label pairs are scored and further clustered to remove residual redundancy.
  • Causal verification: Causal exploitation requires a significant prediction-rate contrast on feature-present counter-evidence examples, verified counterfactual feature removal with semantic preservation, and a significantly negative paired probability shift.The causal criterion jointly requires a negative mean shift below −ε, statistical significance at αcausal, and the correct direction.

4 Experiments

Experiments cover natural language inference and toxicity detection, using BERT and RoBERTa with automated discovery and causal-verification metrics. Debiasing methods consume causally verified features through different retraining or modeling strategies, with multi-seed evaluation and fixed top-k features.

  • Evaluation settings: Experiments evaluate MNLI-trained classifiers on MNLI-matched/-mismatched, SNLI, ANLI R1–R3, and HANS, plus CivilComments-WILDS using 16-cell worst-group accuracy.CivilComments-WILDS contains ≈269k training examples and 8 annotated demographic identities.
  • Models and discovery: BERT-base-uncased and RoBERTa-base are trained with standard empirical risk minimization, while discovery samples N=5,000 balanced training instances per task.GPT-4o generates candidate features and Qwen3-32B evaluates them, separating model families to reduce confirmation bias and preference leakage.
  • Metrics: Discovery and causal verification report odds ratio, coverage, precision, and ¯∆p with paired t-test significance.Sagawa-WGA, proportional |¯∆p| reduction, and Tier A versus Tier C comparisons are reported in appendices.
  • Debiasing methods: DFR retrains only the classification head with ℓ2-regularized logistic regression on programmatically balanced groups while keeping the encoder frozen.DFR-IID adds a random i.i.d. training-data subsample, and DFR-FT extends DFR to full-model fine-tuning.
  • Reproducibility: NLI and CivilComments experiments use three random seeds on both architectures, with top-k SFV features held constant while model-dependent stages and debiasing runs vary.The seeds are 42, 123, and 323; experiments run on 2× NVIDIA RTX A6000 GPUs.

5 Results

UNMASK recovers and causally verifies spurious features across MNLI, CivilComments-WILDS, sentiment controls, and reward-model preference data. Its programmatic groups improve robustness competitively while revealing architecture- and method-specific trade-offs.

  • MNLI feature discovery: UNMASK selects 10 MNLI features and causally verifies 9 on BERT and 6 on RoBERTa consistently across three seeds.Recovered patterns include lexical overlap, negation, contradictory absolutes, and hypothesis length.
  • MNLI feature discovery: RoBERTa is immune to three contradiction features, while the always/every–never/no feature fails on both architectures despite OR = 10.01.The contradiction-feature effects are below ε=0.03 with 95% confidence intervals crossing zero across all three seeds.
  • NLI robustness: 64.99±4.44 HANS accuracy is achieved by BERT PoE-IPW-Group, a +12.58 pp improvement over ERM; RoBERTa gains +4.08 pp.PoE-IPW-Group leads NLI HANS, whereas DFR retains the best in-distribution accuracy and LEACE damages MNLI-m.
  • CivilComments robustness: 71.84±1.94 WGA for BERT Tier A and 72.12±0.26 for RoBERTa Tier C show DFR matching hand-labeled DFR’s 70.1%.Programmatic groups substitute for manual demographic annotation, while DFR reverses its NLI ranking by winning CivilComments.
  • Discovery comparison: UNMASK exceeds PMI+SFV by 3.65 pp WGA while remaining within 2 pp of every baseline on average accuracy.The residual advantage isolates compositional LLM-generated candidates over single-token statistics; UNMASK recovers six of eight canonical identity axes.
  • Causal verification controls: Causal verification returns 0 verified features on sentiment despite mean odds ratios of 4.94 on SST-2 and 5.14 on IMDB.This controlled result indicates that statistical lexical associations alone do not pass the causal gate.
  • Reward-model preference data: RewardBench2 discovery surfaces length, formatting, and refusal-language biases, including apologetic refusals with OR = 161 and precision = 1.00.Nine of ten top Focus features are non-lexical, while Precise_IF yields zero significant features across 95 candidates.

6 Conclusion

UNMASK connects dataset-level statistical artifacts to model-level shortcut exploitation through automated discovery, causal verification, and annotation-free mitigation. Its boolean-expression representation unifies discovery, counterfactual intervention, and group-label generation.

  • 6 Conclusion: UNMASK bridges dataset-level artifacts and model-level shortcut exploitation through three auditable stages: automated discovery, causal verification, and annotation-free mitigation.The pipeline is explicitly organized around these three stages.
  • 6 Conclusion: Boolean expressions provide a single deterministic function for discovering features, performing counterfactual interventions, and generating group labels.This representation is identified as the key design choice.
  • 6 Conclusion: Verification reveals cross-architecture divergence that correlation-only analysis does not expose.The passage contrasts causal verification with correlation-only analysis.

7 Limitations

UNMASK’s executable-boolean-expression representation limits discovery to deterministic surface predicates, excluding latent, semantic, topic, style, and distributional shortcuts. Its causal verification is also limited because counterfactual edits cannot hold every other input property fixed and remove co-firing features.

  • Discovery scope: UNMASK cannot detect latent or semantic shortcuts, topic and style bias, or class imbalance because these cannot be expressed as deterministic input predicates.The limitation follows from requiring every discovered feature to be an executable boolean expression.
  • Discovery scope: The executable-expression requirement bounds what UNMASK can discover to features representable as boolean predicates over inputs.Shortcuts outside this representation remain invisible to the pipeline.
  • Causal verification: Counterfactual causal verification cannot hold every other property of an input fixed while removing the target pattern.The causal stage is therefore bounded by the generator used to produce the edit.
  • Causal verification: Counterfactual edits also remove co-firing features, further constraining causal attribution.This limitation arises during the causal intervention stage.

Reproducibility Statement … A.5 Sentiment Feature Discovery (IMDB and SST-2)

The appendices document reproducible, executable feature discovery and causal verification across MNLI, CivilComments-WILDS, IMDB, and SST-2. The pipeline identifies statistically associated patterns, tests model dependence through counterfactuals, and distinguishes genuine exploitation from noncausal sentiment correlations.

  • Reproducibility Statement: Three fixed seeds (42, 123, 323) support reported NLI and CivilComments results, with the SFV feature set held constant across seeds.Hyperparameters, prompt templates, configuration details, source code, and counterfactual annotation sheets are released.
  • A.1 NLI Spurious Feature Discovery: On MNLI, lexical-overlap variants fire on 26–89% of training examples and produce the largest causal effects, with ¯∆p of −11 to −15 pp.Negation and contradictory absolute terms anchor contradiction predictions, while hypothesis length is associated with neutral predictions.
  • A.2 Pipeline Funnel Statistics: On CivilComments, description-only deduplication reduces 1,695 raw candidates to 506, followed by coverage-aware deduplication from 418 to 223.The second pass removes a further 47% after realized coverage becomes available.
  • A.2 Pipeline Funnel Statistics: MNLI’s funnel distinguishes shared model-independent stages from per-seed model-dependent stages; F150 fails causal verification on both architectures, while F91, F99, and F130 additionally fail on RoBERTa.This leaves 6 verified features on RoBERTa, according to the funnel summary.
  • A.3 Feature Statistics Across Pipeline Stages: The pipeline uses executable boolean expressions as deterministic evaluation backbones, with SFV validation ranking features using odds ratios, coverage, precision, and clustering.All top-k feature odds ratios are BH-corrected significant at p < 0.05.
  • A.4 CivilComments Spurious Feature Discovery: On CivilComments, all 10 selected features are causally exploited by both architectures across all 3 seeds under joint removal, while six contain canonical demographic tokens and three are novel.Post-hoc token matching recovers 6 of WILDS’s 8 canonical identity attributes; single-target verification excludes F120 from Tier A on BERT.
  • A.5 Sentiment Feature Discovery (IMDB and SST-2): 0/8 exploited features are found on IMDB, despite all 10 sentiment-lexicon patterns passing validation with OR ≥4.3 and mean |∆p| of 0.67 pp.Two IMDB features are excluded from testing because of coverage constraints.
  • A.5 Sentiment Feature Discovery (IMDB and SST-2): 0/5 exploited features are found on SST-2, where validated sentiment patterns have OR ≥2.7 and mean |∆p| of 0.26 pp.F116 has OR 14.51 but 0.8% coverage, while shorter SST-2 sentences concentrate features around individual sentiment words.

A.6 Sentiment Ablation: Full Debiasing Grid … C.1 Focus (163 significant, top 10 shown)

The appendix shows that causal verification prevents ineffective sentiment debiasing, while validating counterfactuals, evaluator choices, annotation-free DFR design, implementation details, and RewardBench2 shortcut discoveries. Across tasks, programmatic checks and human audits support reliable feature removal and label preservation, with RewardBench2 revealing interpretable structural preferences.

  • A.6 Sentiment Ablation: Full Debiasing Grid: Seven of eight directly debiased non-causal features stayed within ±0.5 pp of ERM in-distribution, while out-of-distribution accuracy was flat or degraded.LEACE instead destroyed the distributed sentiment representation needed by the task.
  • A.7 Counterfactual Sample Sizes and Test Statistics: All 10 SFV features entered NLI counterfactual generation, but F150 was the sole feature classified as non-exploited on both architectures.Its n=27 accepted pairs yielded post-hoc power = 0.34 at d=0.316, with a confidence interval crossing zero.
  • A.8 Model Design Choices: Qwen3-32B was selected as the unified evaluator because its feature presence agreement was 93.9% versus Gemini Flash’s 86.4%, despite Gemini’s 81.6% versus 80.3% label-preservation advantage.Programmatic boolean expressions were retained for determinism, lower high-volume SFV cost, and avoiding failures in compound and counting categories.
  • A.9 Human Audit of Accepted Counterfactuals: 94/100 (94%) of audited evaluator-accepted counterfactuals preserved gold labels, while feature-removal correctness reached 98/100 (98%).Label preservation was 45/50 (90%) on NLI and 49/50 (98%) on CivilComments; the benchmark figure is a worst-case calibration estimate.
  • A.10 Additional Design Choices: Equal 0.33 weighting across log OR, Coverage, and Precision still selected a rare high-OR feature that causal verification rejected on both architectures.The feature had OR= 10.01 and Cov= 1.0%, illustrating why statistical scoring alone cannot establish model exploitation.
  • B Implementation Details, Hyperparameters and Prompts: The implementation uses ERM fine-tuning followed by frozen-backbone DFR head retraining on programmatically balanced groups, with shared boolean expressions eliminating manual annotation.The LLM pipeline uses configured statistical thresholds and routes calls through OpenRouter except GPT-4o, which uses the OpenAI API directly.
  • B.1 Prompt Templates: Six structured-JSON prompt templates cover pattern discovery, boolean validation and rewriting, counterfactual generation and refinement, and blind label verification.The templates require deterministic, testable boolean logic and preservation of the original semantic relationship while removing spurious patterns.

C.2 Safety (61 significant, top 10 shown) … E Causal Effectiveness of Debiasing Methods

The analysis identifies significant surface patterns across safety, math, factuality, and tie subsets, while Precise_IF yields no significant candidates. Debiasing evaluation shows SCER achieving the strongest BERT worst-group accuracy, with causal-effectiveness analysis defined as an average | ¯∆p| reduction diagnostic over Fcausal.

  • C.4 Factuality (3 significant): Factuality patterns include Markdown section headers, frequent common short words, and prominence adjectives.The highlighted candidates are “Markdown section headers (###)” and “High frequency of common short words (>10%).”
  • C.5 Precise_IF: Zero features reached significance after BH correction across 95 candidate patterns in Precise_IF.No candidate predicate separated chosen from rejected responses, and discovery alone cannot determine whether the subset lacks surface signal or the predicates cannot express it.
  • C.6 Ties (195 significant, top 8 shown): Tie outcomes show length-related rejection signals despite a length-neutral construction, while Harry Potter house names reflect topic-specific annotation noise.Length asymmetry has OR = 2.0, raw word-count differences have OR = 1.9, and the Harry Potter house-name pattern has OR = 3.6.
  • D Sagawa Worst-Group Accuracy on NLI: 73.93±1.74 (+11.86 pp over ERM) is SCER’s highest BERT worst-group accuracy on MNLI dev-matched, while three methods cluster at 77.01% mean WGA on RoBERTa.The hardest cell is NEUTRAL×neg (n=87); SCER also has the tightest standard deviation, 1.99 versus 4.14 for PoE and 4.60 for PoE-IPW-Group.
  • E Causal Effectiveness of Debiasing Methods: The causal-effectiveness diagnostic reports average | ¯∆p| reduction per method over Fcausal for the DFR family and PoE.The diagnostic is specified for BERT-base-uncased across 3 seeds.

F Cross-Architecture Detailed Analysis · G HANS Subcategory Breakdown

Debiasing performance reverses across architectures and tasks: DFR head-only leads CivilComments but fails on NLI, while SCER is comparatively consistent and JTT changes sign on HANS. HANS subcategory analysis shows PoE-IPW-Group leading all three non-entailment categories while preserving entailment accuracy.

  • F Cross-Architecture Detailed Analysis: 71.84 BERT and 72.12 RoBERTa are DFR head-only’s CivilComments WGA, whereas BERT NLI WGA falls to 37.93.SCER and PoE dominate NLI, reflecting heterogeneous syntactic shortcuts across the three label classes.
  • F Cross-Architecture Detailed Analysis: 9.19 pp is RoBERTa’s NLI ERM WGA advantage over BERT, but CivilComments reverses this direction with RoBERTa 3.31 pp lower.The passage attributes the reversal to stronger negation generalization on NLI and deeper demographic shortcut absorption on toxicity data.
  • F Cross-Architecture Detailed Analysis: 0.33 pp separates SCER’s Tier C CivilComments WGA across architectures: 67.24 BERT versus 67.57 RoBERTa.SCER achieves best or tied-best NLI WGA for both architectures.
  • F Cross-Architecture Detailed Analysis: 5.81 pp is JTT’s approximate BERT NLI HANS improvement over ERM, while RoBERTa HANS decreases 2.03 pp, producing a 7.84 pp swing.The passage suggests error-based upweighting amplifies each architecture’s existing ERM error structure.
  • G HANS Subcategory Breakdown: Table 24 reports 3-seed mean±std HANS results for lexical, subsequence, and constituent non-entailment subcategories across both architectures.Non-entailment categories test resistance to specific lexical-heuristic shortcuts; entailment is included for completeness.
  • G HANS Subcategory Breakdown: >97% entailment accuracy across every method other than LEACE confirms that debiasing generally preserves entailment recognition.LEACE is the exception because its entailment performance collapses.
  • G HANS Subcategory Breakdown: PoE-IPW-Group leads all three HANS non-entailment subcategories for both architectures among methods preserving entailment accuracy.Its strongest gains occur on NE-Con, while LEACE’s strong non-entailment scores accompany high variance and substantially degraded overall HANS from entailment collapse.

H Zero-Shot Transfer Results · I Canonical L1 DFR Audit

Zero-shot transfer generally worsens after debiasing, with JTT the main exception and architecture-dependent AG-News effects. Canonical L1 DFR is unstable for BERT but stable for RoBERTa, motivating L2 DFR in the primary experiments.

  • H Zero-Shot Transfer Results: JTT is the only method with consistent positive zero-shot transfer on RoBERTa, while most debiasing methods slightly degrade transfer.The results use NLI-trained models without transfer-task fine-tuning and 3-way P(entailment) scoring.
  • H Zero-Shot Transfer Results: +8.97 pp AG-News transfer occurs for RoBERTa with JTT, versus −12.59 pp for BERT, producing a 21.56 pp cross-architecture swing.Most other debiasing methods slightly hurt AG-News transfer for both architectures.
  • I Canonical L1 DFR Audit: The audited canonical DFR configuration uses an L1 penalty, liblinear solver, StandardScaler, powers-of-10 C-grid from 10−4 to 104, and 20-bootstrap aggregation.This specification produces architecture-dependent seed sensitivity reported per seed in Table 27.
  • I Canonical L1 DFR Audit: 51.51% MNLI-m follows 4 SFV coefficient sign-flips for BERT seed 42, compared with 72.11% after 1 flip at seed 123 and 79.72% after 0 flips at seed 323.The MNLI regression severity tracks sign-flip count, indicating L1 penalizes causally verified shortcut features rather than reweighting them.
  • I Canonical L1 DFR Audit: 14.60 pp MNLI standard deviation across BERT seeds demonstrates severe instability from L1 sign-flips on SFV features.Sign-flips count SFV coefficients changing sign relative to the ERM baseline after L1 head retraining.
  • I Canonical L1 DFR Audit: 0.16 pp MNLI-m standard deviation and zero sign-flips across all seeds show that RoBERTa is fully stable under canonical L1 DFR.RoBERTa CLS embeddings achieve validation accuracy of 0.96-0.98 on group classification, yielding a linearly separable feature space with consistent support vectors.
  • I Canonical L1 DFR Audit: Primary experiments use L2-regularized L-BFGS DFR because it is stable on both architectures.This replaces the audited canonical L1 setup for the main experiments.

J Full CivilComments Results by Feature Tier … M.1 Counterfactual Generation Example

Across CivilComments, open-weight substitution, baseline comparisons, and counterfactual analysis, UNMASK’s benefits depend on feature selection, statistical validation, and causal interventions. The results also identify reproducibility tradeoffs and show that generated counterfactuals can remove multiple spurious patterns while preserving semantics.

  • J Full CivilComments Results by Feature Tier: +1.17 pp: On BERT, DFR Tier A reaches 71.84 versus 70.67 for Tier C by excluding the feature failing single-target verification.Tier C uses all 10 SFV features, whereas Tier A uses the causally verified 9-feature subset; RoBERTa shows no consistent Tier A benefit.
  • K Open-Weight Generator Substitution: The open-weight run recovers the three canonical NLI shortcut classes—lexical overlap, hypothesis-side negation, and length disparity—also recovered by GPT-4o.The comparison reruns discovery through SFV on the same 5,000-sample MNLI input, while causal verification and debiasing are not rerun.
  • K Open-Weight Generator Substitution: +6.4 pp mean precision and +2.03 odds ratio: The open-weight run improves these SFV statistics at slightly lower coverage, with overall scores within 0.018.An independent Qwen3-32B similarity judgement finds 8/10 overlap between the two top-10 feature sets.
  • K Open-Weight Generator Substitution: gpt-oss-120b is recommended as the primary open-weight substitute, but its top-10 omits two compositional cross-field features surfaced by GPT-4o.Those omitted features are described as the axis on which UNMASK’s advantage over single-token discovery rests; DeepSeek-V3 is offered as an alternative.
  • L.1 NLI: UNMASK substantially extends the tightly clustered baselines on HANS’s non-entailment lexical-overlap subset, while LLM-only remains within 0.6 pp of PMI.The comparison uses PoE on BERT-base, and the passage attributes the gap to the statistical validation funnel rather than candidate diversity alone.
  • L.2 CivilComments PoE: Cell-Level Decomposition: 65.96±0.99 versus 69.34±1.09: Under PoE, UNMASK trails PMI on worst-group accuracy while improving all eight non-toxic identity cells and declining on all eight toxic cells.The non-toxic gains range from +0.12 to +6.10 pp, while toxic-cell changes range from −0.90 to −5.86 pp.
  • L.2 CivilComments PoE: Cell-Level Decomposition: Within 0.6 pp: Methods have similar 16-cell average accuracy, so worst-group ordering reflects which cell is worst rather than uniform accuracy differences.UNMASK averages 77.64 versus PMI 78.07, PMI+SFV 78.19, LLM-only 77.97, and ERM 77.67; dispersion falls for all debiased methods from ERM’s 14.18.
Loading 2608.09209v1…