PaperlayerView exact revision on arXiv

Source-linked AI summary

Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering

Hyojung Han, Jongmin Kim, Seung-Hun Jeon

arXiv:2610.19343v1cs.LGcs.CL

TL;DR

A gate-versus-random comparison cannot distinguish within-support selection from redistribution across supports. CGSD constructs four budget-matched arms to separate those mechanisms and their interaction, subject to finite-pool feasibility. In retrieval, within-support selection is the only component reported as exceeding its calibrated null, while redistribution and interaction remain unresolved at available power.

  • Problem

    A gate changes both which examples survive within input regions and how training mass is distributed across regions, so a gate-versus-random comparison cannot isolate these mechanisms.

    Source: §1 · p. 1

  • Method

    CGSD crosses gate eligibility with source-preserved or gate-induced support distributions in four budget-matched arms, decomposing the gate effect into selection, redistribution, and interaction.

    Source: §3 · p. 3

  • Results

    +27.3 items for within-support selection under the model-confidence threshold; redistribution and interaction are unresolved, and retrieval point estimates are dominated by selection.

    Source: §7 · p. 6, §10 · p. 9

  • Takeaways & Limitations

    Making arm existence explicit distinguishes a decomposition from a constructible-but-underpowered design or a non-constructibility result, preventing redistribution from being read as sample-level gate quality.

    Source: §10 · p. 9

    The retrieval evidence covers two gate families and locales but only one backbone, endpoint, and data family, so it does not demonstrate domain generality.

    Source: §9 · p. 8

Abstract

from arXiv · show

A training-data quality gate changes both which examples survive within regions of the input space and how training mass is distributed across those regions, so a gate-versus-random comparison does not isolate these mechanisms. We introduce Counterfactual Gate-Support Decomposition (CGSD), a four-arm, budget-matched design that crosses gate eligibility with support distribution and decomposes the observed gate effect into within-support selection, between-support redistribution, and their interaction. The practical constraint is finite-pool realizability: we characterize mass and token-assembly conditions under which the required arms can be constructed, together with a support-extinction case in which the ordinary design is undefined. On Korean-English and Korean-Japanese retrieval, the mean within-support selection component is the only one that exceeds its calibrated null band (+29.7 and +5.6 holdout items; 30/30 positive replicates), decisively in Korean-English and marginally in Korean-Japanese; redistribution and interaction remain unresolved at the available power. The same pattern holds for a second, unrelated gate family -- a model-confidence threshold rather than a deterministic verifier -- judged against its own null (+27.3 items, 9/9 positive), so the result is not a property of one gate's mechanism. On a mathematics gate, zero-eligibility supports leave the ordinary design undefined at every budget, while the extended design is constructible only inside narrow budget bands that an exhaustive scan locates and that the natural operating point misses -- the assembly condition separating a property of the pool from a limitation of the packer. CGSD therefore returns a decomposition, a constructible-but-underpowered design, or an explicit non-constructibility result, and says which; it prevents support redistribution from being silently interpreted as sample-level gate quality.

1 Introduction

A gate-versus-random comparison mixes within-support selection with redistribution across regions, so it cannot identify which mechanism produced the downstream difference. CGSD constructs a budget-matched contrast and makes the finite-pool conditions for constructing it explicit.

1 Introduction

  • The gate-versus-random comparison combines selection within regions and redistribution across them, so it cannot isolate either mechanism.A gate can remove very different fractions of different topics.Source: §1 · p. 1
  • CGSD crosses eligibility and support distribution in four training sets at the same token budget, decomposing the observed gate effect into three terms.The arms differ only in the factor being varied, and the decomposition holds replicate by replicate.Source: §1 · p. 1
  • The decomposition is a familiar 2 × 2 factorial identity; the contribution is operationalizing it for finite pools at fixed exposure and characterizing when its arms cannot be realized.The authors do not claim the algebra itself as new.Source: §1 · p. 1
  • Finite-pool feasibility requires both sufficient support mass and token-size assembly; zero eligible mass, quota tiling, and packer failure are distinct cases.The prescriptions differ: a smaller budget can fix insufficient mass, but cannot fix a support with no eligible examples.Source: §1 · p. 2
  • The paper pairs the operational contrast with calibration, analytic recovery, and infeasibility controls, then applies it to retrieval and audits mathematics and text-to-SQL pools.The feasibility audit distinguishes extinction, empty feasible regions, and a narrow feasible band.Source: §1 · p. 2

2 Related work

Prior work studies better filtering rules, mixture control, example attribution, decomposition, and experimental rigor. CGSD is diagnostic: it takes a gate as given and separates its mechanisms under finite-pool constraints.

2 Related work

  • Filtering methods propose which examples to keep, including quality filters, deduplication [1] [16], pruning [24], task-targeted selection [27], and datamodel-driven selection [7]; CGSD instead diagnoses a given rule.Controlled curation benchmarks also make the comparison itself an object of study.Source: §2 · p. 2
  • Mixture research shows that changing proportions can affect downstream performance independently of individual-example choice, making CGSD's separation of the two levers relevant [28], [8], [17] [30], [29], [5].This line includes domain reweighting, mixture search, and importance resampling.Source: §2 · p. 2
  • Example-attribution methods assign value to individual training examples, often without retraining [11] [15], [22], [14], [10] [26]; CGSD retrains constructed pools to distinguish mechanisms, including for opaque gates.Its attribution is coarser-grained and directed at mechanisms rather than examples.Source: §2 · p. 2
  • CGSD's mass condition parallels positivity in matching and propensity methods [25], while pooled-arm training raises interference concerns studied in causal inference.The related decomposition precedents include Oaxaca–Blinder and mediation.Source: §2 · p. 2
  • The validation protocol follows established work on seed variance and interval reporting [6] [12] [20], adding null bands tied to the exact measured configuration.A threshold cannot be silently reused across operating points.Source: §2 · p. 2

3 The estimand

The estimand compares four support-controlled training arms built from pre-gate support labels and fixed token budgets. Its three components separate within-support selection, support redistribution, and their interaction, while E is not a general measure of example quality.

3 The estimand

  • Each source example has a token count, gate verdict, and support label assigned by a partition of the pre-gate input.Support labels must be pre-gate; otherwise redistribution would be defined using the treatment and become circular.Source: §3 · pp. 2–3
  • The four arms cross eligibility with support shares: unrestricted or gate-eligible examples, each using source-preserved or gate-induced shares.Per-support budgets are filled by sampling to token targets.Source: §3 · p. 3
  • The fifth arm, R, is an unconstrained random reference at the same token budget, allowing the effect of imposing support control to be reported separately.The reported reference contrast is Y(D00) − Y(R).Source: §3 · p. 3
  • The support-controlled arms decompose the gate contrast exactly into E, S, and I for each replicate.The decomposition identity is by construction, not merely in expectation.Source: §3 · p. 3
  • E measures restricting examples to gate-eligible ones while holding support distribution fixed; it is not a general measure of quality or why eligibility helps.Label correctness, length, formatting, and annotator agreement all fall within E.Source: §3 · p. 3
  • E, S, and I are conditional on budget and support partition, so estimates at different operating points are different estimands.Absolute magnitudes are not comparable across pools with different budgets or holdout sizes.Source: §3 · p. 3

4 Feasibility: two independent conditions

The four-arm design is constructible only when both eligible mass and per-support token assembly satisfy the matching contract. Feasibility also depends jointly on budget and support partition, and assembly feasibility need not improve as the budget changes.

4.1 Mass

  • D10 is mass-feasible only when B/T ≤ min_c e_c; a negative mass margin is repaired by lowering B.Source: §4.1 · p. 3

4.2 Assembly

  • Because examples are indivisible and arms draw from different eligibility pools, assembly is checked per arm; G_as ≥0 means an in-tolerance assembly exists, not that a packer finds it.Source: §4.2 · p. 3
  • Assembly feasibility is non-monotone in B, while τ = 0.5% bounds realized support drift to 0.503%.Lowering B does not reliably repair assembly and can make it worse.Source: §4.2 · pp. 3–4

4.3 The feasible region

  • Feasibility depends jointly on budget and partition; refining the Korean–English partition from K=8 to K=16 moves B/T=0.40 outside the feasible region.M binds throughout this pool while G retains +51 to +504 tokens of slack.Source: Figure 1 · p. 4, §4.3 · p. 4

5 Support extinction

A positive-mass support with no gate-eligible examples makes the ordinary support-preserving counterfactual infeasible at every positive budget. The extended decomposition reports this support-extinction contrast alongside the other components.

5 Support extinction

  • If a source support has positive mass and zero gate eligibility, the ordinary design is infeasible for every positive budget; unlike M < 0, lowering B cannot repair it.Source: §5 · p. 4
  • The extended design drops extinct supports and renormalizes the survivors’ original proportions at the same budget, with E, S, and I defined over surviving supports.Source: §5 · p. 4
  • XB is mandatory in the extended decomposition: omitting it changes the estimand, while its value also reflects the replacement policy used for extinct mass.The stated policy returns extinct mass to survivors in their original proportions; a different replacement policy yields a different XB.Source: §5 · pp. 4–5

6 Validation before use

The validation checks construction and estimation through calibration, known-effect recovery, and frontier stress tests. The reported safeguards also bind verdicts to their exact configuration and qualify what the component pattern can establish.

6 Validation before use

  • Under same-rate random masks, each component’s decision threshold is δ = q95(|null|); the applied threshold is 2.8–3.0× matched aggregation, and no multiplicity correction is applied across six tests.The interval-separation diagnostic does not override the pre-specified rule, and the paper reports no family-wise error rate for the overall pattern.Source: §6 · p. 5
  • Analytic planted-effect tests had |bias| ≤0.18 items and nominal 95% interval coverage of 0.927–0.960; learner-in-loop interventions recovered the planted component and sign.Source: §6 · p. 5
  • At the tightest mass margin, bias, RMSE, and coverage were unchanged, but the constructible support effect was under one holdout item.Source: §6 · p. 5
  • Each null threshold is bound to a configuration signature covering locale, partition, budget, holdout, seeds, gate, pass rate, endpoint, and constructor hash.A prior mismatch between calibration and operating conditions triggered withdrawal and recalibration of both locales.Source: §6 · p. 5

7 Retrieval: the effect is within-support

In both retrieval locales, within-support selection is the only component whose mean clears its calibrated null band; the Japanese result is marginal under an interval-based reading. A second gate family shows the same qualitative component pattern, while redistribution and interaction remain unresolved at the available power.

Experimental setup

  • The retrieval evaluation uses K = 8 k-means supports on pre-gate query embeddings and top-1 leaf accuracy on a held-out split.Each locale runs 10 selection seeds × 3 training seeds, with hierarchical-bootstrap confidence intervals.Source: §7 · p. 6

Primary retrieval result

  • Within-support selection is the only component whose mean clears its calibrated null band in either locale; EN meets conservative separation, while JP passes only the point-estimate rule and is marginal under the interval-based reading.Source: §7 · p. 6, Table 1 · p. 7

Second gate family

  • The confidence-threshold gate yields E = +27.3 items with its interval clearing δ’s upper end, while S and I remain unresolved against its own null.It passes 60.3% of the pool versus 47.9% for the verifier, so the results are not an effect-size comparison.Source: §7 · p. 6

Redistribution estimate

  • The descriptive redistribution composite R = S + I is −0.5 items for EN and −0.4 for JP, and inherits the power limits of its components.It was not pre-specified.Source: §7 · p. 6

Partition sensitivity

  • Across K = 4, 8, and 16, E stays positive but changes magnitude; K=16 minus K=8 is −6.3 items [−12.4, −0.7].The difference appears in ΔG rather than S; replacing D00 with the unconstrained reference shifts the estimate by −0.5 items [−5.1, +3.5].Source: §7 · pp. 6–7

8 Feasibility audits: extinction, narrow bands, and empty regions

The mathematics audit finds structural extinction that makes the ordinary design undefined, while the extended design is constructible only in narrow budget bands. Text-to-SQL feasibility also varies by pool, and available pool sizes do not support effect estimates.

Mathematics pool

  • Two of four mathematics supports have zero eligible mass, making the ordinary four-arm construction undefined at every positive budget.The extinct supports contain 6.1% of tokens; the token share is reported because exposure is matched in tokens.Source: §8 · p. 7, Table 2 · p. 8
  • In mathematics, extinct supports account for 11.9 of 13.0 tokens of Q, and their coarse token lattices bind assembly in this pool.The authors caution that extinction need not cause the same assembly issue when the extinct supports are large.Source: §8 · p. 7

Text-to-SQL pools

  • The five text-to-SQL pools have no extinction, but four have empty feasible regions and the fifth is limited by global assembly failures despite exact within-support solving.The largest pool also lacks a power-safe operating point under the pre-fixed interior margin.Source: §8 · pp. 7–8
  • Four of five text-to-SQL pools have no budget with G ≥0 for every arm; the fifth is constructible at 68 of 86 tested budgets.The realized band spans B/T ∈[0.309, 0.643]; 18 certified budgets still fail at the global assembly stage.Source: §8 · p. 7

Scope of the audits

  • These pools establish feasibility conditions, not effects: the mathematics and live-clean pools are far smaller than one retrieval arm and cannot support informative component estimates.A retrieval arm uses 95,162 tokens versus 3,915 for the mathematics pool and 15,042 for live-clean; the mathematics estimate would have an interval wide enough to contain any conclusion.Source: §8 · p. 8

Mathematics pool

  • At B = 1,989, the survivor arms assemble but Dall 00 has G = −3; exhaustive scanning finds 54 feasible budgets in B/T ∈[0.480, 0.495].The pre-registered operating point falls in a gap between feasible bands; moving 50 tokens down to B = 1,939 yields G = +0.7 for Dall 00.Source: §8 · p. 8, Table 2 · p. 8
  • The mathematics pool’s narrow bands distinguish a pool-level assembly condition from a packer failure: 49 of 55 attempted budgets build after switching to an exact assembler.The budget B = 1,989 reports granularity, while five other failures are attributed to the packer.Source: §8 · p. 8

9 Limitations

The retrieval evidence is limited in scope and power, and unresolved components should not be read as zero. The decomposition is also conditional on its operating point and matching contract.

Evidence scope

  • Retrieval evidence covers two locales and two gate families, but only one backbone, endpoint, and data family.It is replication across gates and languages, not a demonstration of domain generality.Source: §9 · p. 8

Power and interpretation

  • The second gate uses three selection seeds against ten, leaving its selection-variance component estimated at zero because training noise dominates and the estimator truncates.Unresolved components indicate limited power, not zero effects.Source: §9 · p. 8
  • The retrieval results do not establish that the same gate helps retrieval and hurts mathematics, because the mathematics evidence is about feasibility rather than a negative effect.E also remains uninterpreted as to mechanism.Source: §9 · p. 8

Matching contract

  • FΠ is defined relative to the matching contract, so changing that contract changes the feasible region.The authors regard changing it after seeing a failure as out of bounds.Source: §9 · pp. 8–9

10 Conclusion

CGSD separates within-support selection from support redistribution, but its four-arm design depends on whether the arms can be constructed. The audits show why feasibility and power must be considered before interpreting component estimates.

Findings

  • On retrieval, within-support selection dominates the point estimates, while redistribution remains unresolved at available power—not shown to be zero.The conclusion identifies extinction and narrow-band constructibility as the honest mathematics-audit result, rather than an effect estimate.Source: §10 · p. 9

Feasibility

  • Support extinction leaves the ordinary comparison undefined at every budget, while the extended design is constructible only in narrow bands missed by the pre-registered operating point.Four of five text-to-SQL pools have empty feasible regions; the fifth has a narrow band.Source: §10 · p. 9

Implication

  • The paper argues that feasibility results are needed before practitioners can trust a decomposition, even when they cannot act on those results directly.Source: §10 · p. 9

Availability

The implementation and result-generation pipeline are released, but the source pools and evaluation splits are not. Reproducing the training arms therefore requires equivalent data.

Released materials

  • The implementation is released under Apache-2.0 with recorded result files and scripts that regenerate the paper’s figures, tables, and numbers.The constructor, feasibility tests, and extinction split are included with data-free gate harnesses.Source: Availability · p. 9

Data access

  • The source pools and evaluation splits are not redistributed, so retraining the arms requires equivalent data of the reader’s own.Source: Availability · p. 9

Reimplementation

  • The estimand, decomposition identity, feasibility conditions, extinction proposition, and drift bound are specified in Sections 3–5 for reimplementation.Source: Availability · p. 9

Related papers

Loading 2610.19343v1…