PaperlayerView exact revision on arXiv

Source-linked AI summary

When Audit Quality Fails to Predict Downstream Utility: A Counterfactual Study of Synthetic-Data Selectors for Low-Resource African NLP

Son Ha Xuan, Phat T. Tran-Truong, Xuan-Bach Le

arXiv:2609.18960v1cs.CL

TL;DR

Synthetic-data selectors rely on the premise that judged quality identifies examples that help downstream learning. In a controlled replay across African-language classification tasks, the audit and downstream rankings diverge. The results support reporting both kinds of metrics on the same retained sets within this study’s controlled scope.

  • Problem

    Synthetic-data pipelines must choose among more candidates than a downstream learner can absorb, and quality-ranking methods assume strong scorers identify examples worth keeping.

    Source: §1 · p. 1

  • Method

    The study replays five matched-budget selectors across eight task-language cells using shared audit records and compares audit metrics with downstream Macro-F1.

    Source: §1 · p. 1

  • Results

    Audit and downstream rankings diverged across the eight cells; within-cell judged-label-correctness–Macro-F1 Spearman averaged ρ=0.04 (median 0.00).

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    In this controlled setting, audit quality describes the selected pool but does not guarantee downstream utility, so evaluation should report both metrics on the same retained sets.

    Source: §7 · p. 8

    The replay covers eight task-language cells at one seed, and judged correctness and quality depend on a single multilingual judge prompted in English.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Quality-aware synthetic-data selection rests on a proxy: examples that an LLM judge rates as good should also help a downstream model learn. In a controlled replay in low-resource African-language classification, we show that this proxy breaks. Across four languages (Amharic, Hausa, Swahili, Yoruba), two classification tasks (MasakhaNEWS, AfriSenti), and five matched-budget selectors, audit rankings and downstream rankings diverge. Within each cell, the Spearman between judged label correctness and Macro-F1 across selectors has mean $\rho{=}0.04$ (median $0.00$), showing that the mismatch is not an aggregation artifact. \method{}-V2, our counterfactual audit framework, produces the cleanest selected pool on three audit channels at once: highest judged label correctness ($0.904$ vs.\ $0.767$ for naive, a $17.9\%$ relative gain), lowest shortcut score, and a hard-reject rate of $0.162$ vs.\ $0.486$ for naive. AlpaGasus nevertheless leads downstream Macro-F1 ($0.202$ vs.\ $0.163$ for \method{}-V2), and the inversion persists on the five non-degenerate cells. The lesson is methodological: in this controlled setting, audit quality is a property of the selected pool, not a guarantee of downstream utility. Synthetic-data evaluation should therefore report audit and downstream metrics on the same retained sets. We release the audit tables, per-selector retained pools, and a claim ledger that links every reported number to its source row.

1 Introduction

Synthetic-data selectors assume that scoring can identify examples worth keeping, but this controlled replay finds that audit rankings and downstream rankings disagree. The study compares five matched-budget selectors across eight African-language classification cells and reports both audit and downstream metrics on the selected sets.

1 Introduction

  • The replay covers eight task-language cells across MasakhaNEWS and AfriSenti, comparing five selectors on the same audit record and at matched budgets.It uses one generator and one downstream backbone, framing the findings as a controlled case study rather than a general law.Source: §1 · p. 1
  • Within-cell Spearman between judged label correctness and Macro-F1 averaged ρ=0.04 (median 0.00), while downstream rankings reordered the selectors.The disagreement also appeared on non-judge channels; on five non-degenerate cells, CoSDA-V2 reached 0.163 while AlpaGasus reached 0.226.Source: §1 · pp. 1–2
  • CoSDA-V2 leads judged label correctness at 0.904 versus 0.767 for naive selection, with a 17.9% relative gain, and has the lowest shortcut score.All four quality-aware selectors beat naive on judged label correctness, judged quality, and hard-reject rate.Source: §1 · p. 1
  • The study formalizes five per-example audit channels and a calibrated hard-reject decision, then reports audit and downstream metrics on the same selected sets.It releases per-candidate audit records, retained pools, and a claim ledger linking reported numbers to replay-table rows.Source: §1 · p. 2

2 CoSDA: A Counterfactual Audit Instrument

CoSDA is presented as a measurement instrument for inspecting synthetic candidate pools before training. Its per-candidate audit record supports matched-selector comparisons, including a budget-preserving CoSDA-V2 reranker.

2 CoSDA: A Counterfactual Audit Instrument

  • CoSDA makes selected-pool contents inspectable before training, and comparison selectors read the same per-candidate audit record.The audit trace links table cells and claims back to replay rows and underlying candidate records.Source: §2 · pp. 2–3
  • Task-specific counterfactual pairs flip a topic or sentiment cue while holding language, register, and other content fixed; usable counterfactual rates vary by language.Extending the validator beyond classification requires defining the corresponding task-specific transformation.Source: §2 · p. 2
  • Each candidate receives five normalized audit scores: utility, leakage, shortcut, diversity, and counterfactual consistency.Utility combines generator confidence, teacher agreement, and a validation-neighbour vote; leakage uses overlap and similarity checks, while diversity measures distance from existing examples.Source: §2 · pp. 2–3
  • The audit record also stores judged label correctness and judged quality, while reported audit metrics are averaged over each selector’s 64-example retained pool.Hard-reject rate measures residual threshold violations in the retained pool, not the pre-filter reject rate.Source: §2 · p. 3
  • CoSDA-V2 hard-rejects candidates, then ranks survivors by a weighted score under class-balance, source-seed, and budget constraints shared with AlpaGasus and DEITA.The comparison includes naive selection, judged-quality ranking, quality-plus-complexity-plus-diversity ranking, and CoSDA-EQ’s hard-reject-then-random-fill procedure.Source: §2 · p. 3
  • Leakage threshold τL and score weights are calibrated on a held-out Setswana split, while τH and τC follow fixed protocol choices.All threshold groups remain identical across the eight reported cells.Source: §2 · p. 3

3 Experimental Setup

The experiment is a deterministic, matched-pool replay of five selectors across two classification tasks and four African languages. It compares downstream Macro-F1 and audit metrics on the same retained synthetic sets.

3 Experimental Setup

  • The eight cells cover MasakhaNEWS news-topic and AfriSenti sentiment classification in Amharic, Hausa, Swahili, and Yoruba.Macro-F1 is the downstream metric, and each task-language cell is evaluated independently.Source: §3 · p. 3
  • Each cell uses 64 gold examples and a 3× synthetic budget, yielding 64 retained synthetic examples per selector from a shared candidate pool.The deterministic replay uses seed 13, making this a within-pool test of selection criteria rather than independently sampled generations.Source: §3 · pp. 3–4
  • Five selectors use identical class-balance and source-seed constraints: naive, AlpaGasus-style, DEITA-style, CoSDA-EQ, and CoSDA-V2.A Gold-only reference is reported in the appendix and excluded from the main comparison because it uses no synthetic data to audit.Source: §3 · p. 4
  • Each classifier trains on the same 64 gold seeds plus a selector’s 64 retained synthetic examples, and evaluation reports Macro-F1 alongside six selection-quality metrics.Those metrics are judged label correctness and quality, counterfactual consistency, leakage, shortcut, and hard-reject rate.Source: §3 · p. 4
  • Numeric claims are computed from the full per-cell replay table, and the released claim ledger maps table cells and aggregates to source rows.Appendix A lists the ground-truth artifacts.Source: §3 · p. 4

4 Results: The Quality-to-Utility Gap

Across the eight task-language cells, audit rankings and downstream Macro-F1 rankings diverge: AlpaGasus leads downstream while CoSDA-V2 leads judged label correctness. The mismatch persists within cells and under robustness checks, although the study interprets it as rank disagreement in a controlled setting.

4.1 The gap

  • +0.036 Macro-F1 over naive goes to AlpaGasus, while CoSDA-V2 leads LC; DEITA, CoSDA-EQ, and V2 remain below naive despite better audit profiles.The selector-level LC–Macro-F1 Spearman is ρ=0.10 across five selectors; the paper treats that statistic qualitatively and prioritizes within-cell analysis.Source: §4.1 · p. 4, Table 1 · p. 4
  • CoSDA-V2 leads on LC and shortcut H, CoSDA-EQ on C and hard-reject, and AlpaGasus marginally on Q; only AlpaGasus also gains downstream over naive.The four quality-aware selectors separate from naive on LC, Q, C, and hard-reject; non-judge C and composite-channel correlations with Macro-F1 are both ρ=−0.20.Source: §4.1 · p. 5, Table 2 · p. 5

4.2 Head-to-head: LC leader vs. downstream leader

  • CoSDA-V2 leads AlpaGasus on LC, hard-reject, and C, trails by 0.003 on Q, yet AlpaGasus has a +0.039 mean Macro-F1 lead.The selectors tie in three cells, with three V2 wins and two AlpaGasus wins; the downstream edge reflects effect size rather than broad dominance.Source: §4.2 · p. 5, Table 3 · p. 5
  • +0.04 mean and 0.00 median within-cell LC–Macro-F1 Spearman, with four positive and four negative cells, show the mismatch is not only an aggregation effect.Figure 1 presents the within-cell correlations across the five selectors.Source: §4.2 · p. 5, Figure 1 · p. 5

4.3 Robustness

  • −0.004 aggregate V2-minus-naive Macro-F1 comprises two positive cells and six negative or near-zero cells; three cells are near-degenerate.The positive cells are Swahili news (+0.129) and Hausa sentiment (+0.206); the near-degenerate cells are Yoruba news, Swahili sentiment, and Yoruba sentiment.Source: §4.3 · pp. 5–6, Figure 2 · p. 5, Table 4 · p. 6
  • Removing Hausa sentiment shifts the eight-cell V2−naive mean from −0.004 to −0.034, consistent with cell effects of opposite signs rather than a uniform advantage.Figure 2 displays per-cell V2-minus-naive Macro-F1 deltas.Source: Figure 2 · p. 5, §4.3 · p. 6, Table 5 · p. 6
  • On five non-degenerate cells, AlpaGasus averages 0.226 Macro-F1 versus 0.163 for V2; LC-to-F1 Spearman rises to +0.50, and the inversion persists.Cell-bootstrap intervals for both selectors’ mean deltas over naive include zero; the paper treats audit-versus-downstream disagreement as the directly testable signal.Source: §4.3 · p. 6

5 Analysis: Why the Audit Misses

The analysis traces audit–utility disagreement to three mechanisms: strict filtering can remove useful support, quality-ranked examples can carry filtered topical or decision-boundary signal, and judge scores can saturate. Cell cases and selector comparisons motivate reporting audit and downstream metrics on the same retained sets.

5 Analysis: Why the Audit Misses

  • In V2’s five cells losing to naive, residual hard-reject averaged 0.23, versus 0.08 in its two winning cells; its correlation with the V2−naive Macro-F1 delta was −0.56 (p=0.15).The authors call the per-cell direction suggestive in this eight-cell trace, not decisive.Source: §5 · p. 6
  • AlpaGasus’s diversity score was 0.590, below naive (0.593) and CoSDA variants (0.603–0.604), while DEITA was highest at 0.649.The authors identify topical or decision-boundary support in quality-ranked candidates filtered by V2 as the remaining explanation, not variety preservation.Source: §5 · pp. 6–7
  • Judged label correctness was at least 0.77 for every selector and saturated around 0.90; judged quality ranged from 0.66 to 0.75, limiting their resolution for downstream Macro-F1.The disagreement also appeared on non-judge channels C, H, and hard-reject.Source: §5 · p. 7
  • V2’s audit advantages do not guarantee downstream gains: strict filtering may remove classifier support, AlpaGasus’s lead is not explained by diversity, and judge scores saturate.The analysis treats audit channels as complementary evidence about selected pools, not substitutes for downstream validation.Source: §5 · pp. 6–7

5.1 Cell-level case studies

  • Across cells, V2’s largest losses coincided with high residual hard-reject rates, while its low rates accompanied gains; three near-degenerate cells had five of six baselines tied.Amharic had a 33.4% counterfactual pass rate, compared with 52%–59% elsewhere; the groups align with the naive Macro-F1 scale.Source: §5.1 · p. 7

5.2 What the audit pairs with

  • Report audit and downstream metrics on the same selected sets, and treat judged label correctness as an audit channel rather than a downstream proxy.The replay reports a selector-level Spearman of ρ=0.10 for judged label correctness; V2 is preferred when leakage, shortcut robustness, or public-facing label correctness carries compliance weight.Source: §5.2 · p. 7
  • CoSDA-EQ and V2 reached similar diversity (0.604 vs. ≈0.60) but different hard-reject rates (0.109 vs. 0.162); the proposed follow-up is to graft V2’s hard-reject onto AlpaGasus’s ranking.Their downstream means were within each other’s bootstrap uncertainty.Source: §5.2 · p. 7

6 Related Work

Related work covers synthetic supervision and selectors that rank generated examples, while noting that selection quality and downstream utility need not align. It also situates the study among data-centric diagnostics, counterfactual tests, and African-language classification benchmarks.

Related work

  • Synthetic supervision has been generated from seeds or task descriptions, with multilingual extensions; selectors rank examples using quality, complexity, diversity, or validation influence.Examples include AlpaGasus (Chen et al., 2024), DEITA (Liu et al., 2024), LESS (Xia et al., 2024), and QuRating (Wettig et al., 2024).Source: §6 · pp. 7–8
  • LLM-judged label correctness certifies that a candidate maps to its intended label, not that a small classifier will learn a better decision boundary from it.Each retained pool in this setting has 64 synthetic examples, where duplicated, shortcut-heavy, or coverage-expanding examples can matter.Source: §6 · p. 8
  • Prior work combines quality with complexity and diversity, selects by validation influence, and documents cases where cleaner subsets underperform noisier supersets.The replay makes the misalignment explicit at selector level using within-cell rank statistics on identical candidate pools.Source: §6 · p. 8
  • Data-centric diagnostics study useful or harmful examples, while counterfactual and behavioral tests probe task-relevant responses rather than artifacts; this work applies such diagnostics to synthetic-data selection.The audit includes leakage, shortcuts, and counterfactual consistency before testing whether those signals predict Macro-F1.Source: §6 · p. 8
  • The replay uses MasakhaNEWS and AfriSenti across four languages spanning Afro-Asiatic and Niger-Congo families and Ge’ez and Latin scripts.The passage identifies task-specific counterfactual validators as necessary for extending the audit to sequence labelling, intent/slot filling, or summarisation.Source: §6 · p. 8

7 Conclusion

The replay separates audit quality from downstream utility and finds that the two can rank selectors differently. Its conclusion is diagnostic and limited to this controlled setting, with broader applicability presented as a conjecture.

Conclusion

  • In this controlled replay, CoSDA-V2 leads judged label correctness but trails downstream, while AlpaGasus reverses that ordering; robustness checks preserve the gap.The authors recommend reporting audit and downstream metrics on the same retained sets.Source: §7 · p. 8
  • The authors conjecture, but do not test, that this decoupling may also arise in filtered pretraining, instruction-tuning curation, and preference-data pipelines.Released traces are intended to support broader replay.Source: §7 · p. 8

Limitations

The replay’s conclusions are bounded by its classification-only, single-seed design and by choices in judges, detectors, generator, backbone, and selector coverage. Its downstream claim is rank disagreement, not a selector-versus-naive win or a causal account of the mechanisms.

Limitations

  • The eight-cell replay uses a single seed, and bootstrap confidence intervals for V2−naive and AlpaGasus−naive both contain zero.The authors therefore focus the downstream claim on rank disagreement rather than a selector-versus-naive win.Source: Limitations · p. 9
  • The completed study covers two classification tasks; sequence labelling, intent/slot filling, and summarisation require task-specific counterfactual validators.These task extensions are not reported as completed results.Source: Limitations · p. 9
  • Judged label correctness and quality use one English-prompted multilingual LLM judge, so a multi-judge or multi-prompt sweep could change the LC ordering.The reported LC and Q values are properties of that judge and prompt, although disagreement also appears on non-judge channels.Source: Limitations · p. 9
  • Leakage checks target observable contamination rather than upstream-training memorisation, while the shortcut probe uses a fixed feature inventory.Because selectors share the audit fields, detector imperfections affect interpretation of audit-quality differences uniformly.Source: Limitations · p. 9
  • All candidates come from Qwen2.5-14B-Instruct under one nucleus-sampling configuration, and downstream models use XLM-RoBERTa base.The headline finding is scoped to this generator/backbone pair; a stronger pair could re-align rankings.Source: Limitations · p. 9
  • Score weights were calibrated on a held-out Setswana split, while a ±10% sensitivity sweep on thresholds and weights remains proposed rather than completed.The replay reuses the calibrated weights and fixed protocol thresholds across cells.Source: Limitations · p. 9
  • The matched-budget selectors reimplement selection criteria rather than original codebases, omit validation-aware selectors such as LESS, and explain mechanisms without causal ablations.A validation-aware selector is identified as the most relevant missing comparison, and a controlled intervention is proposed to isolate audit-step effects.Source: Limitations · p. 9

Ethical Considerations

The paper frames synthetic data as a possible way to ease annotation pressure while recognizing risks to cultural context, language use, and privacy. Its audit metrics are automatic rather than human-validated, so deployment should include target-language community review.

Ethical Considerations

  • Synthetic data can ease annotation pressure but may distort cultural context, normalise incorrect language use, or expose private information from web-trained generators.CoSDA logs provenance, checks leakage at three levels, and measures counterfactual consistency before examples enter the retained pool.Source: Ethical Considerations · pp. 9–10
  • Released artifacts record dataset licenses, mark generated examples, and include a data statement, datasheet, and removal procedure for individuals identifying non-consensual content.The accompanying documentation lists language coverage, intended use, and known failure modes.Source: Ethical Considerations · p. 10
  • Automatic audit metrics do not establish human-validated cultural plausibility, so deployment should include review by target-language speakers, especially for political news and sentiment data.Source: Ethical Considerations · p. 10

A Reproducibility Checklist

The appendix makes the replay auditable by linking reported claims to shared artifacts and exposing the prompts, audit records, and detail tables behind the main results. It also distinguishes completed evidence from deferred validation and extensions.

A Reproducibility Checklist

  • Every numeric claim derives from one per-cell replay table, and the released claim ledger links reported values to their source rows or aggregate statistics.The table records retained-set size, audit channels, hard-reject rate, and downstream Macro-F1; Table 6 identifies authoritative artifacts.Source: A Reproducibility Checklist · p. 13, Table 6 · p. 14
  • The release includes manifests, candidate examples and audit records, per-cell results, and scripts that let readers recompute selectors from candidate-level evidence.Candidate-level records are separated from selected-set summaries so readers can recompute selections without relying on aggregate tables.Source: A Reproducibility Checklist · p. 13
  • Thresholds and weights remain fixed across all eight cells, with τL and weights calibrated on held-out Setswana data and τH and τC fixed by protocol.Multi-seed intervals, human annotation, generator replacement, budget sweeps, and task extensions are not completed results.Source: A Reproducibility Checklist · p. 14

B Prompts and Audit Protocol

  • The protocol stores judge-facing quality fields and non-judge diagnostics for candidates, while downstream classifiers train only on the examples each selector retains.Selectors receive the same candidate pool within each task-language cell, enabling comparison without different generation runs.Source: B Prompts and Audit Protocol · p. 14
  • Counterfactual edits change one classification-relevant variable while preserving language, register, and as much non-label content as possible, then score whether the expected label flip occurs.Source: B Prompts and Audit Protocol · p. 14
  • Recorded automatic fields include judged correctness and quality, counterfactual consistency, leakage, shortcut, hard-reject status, and selected-set membership; they are not human annotations.Table 7 illustrates kept and rejected categories, while a speaker-validation study is proposed but not reported as completed evidence.Source: B Prompts and Audit Protocol · pp. 14–15, Table 7 · p. 15

C Detail Tables and Plots

  • The detail tables provide downstream Macro-F1 including a gold-only reference, the full audit-channel view including leakage, and remaining per-cell Macro-F1 results for DEITA and CoSDA-EQ.The gold-only reference is excluded from audit-versus-downstream rank comparisons because it has no retained synthetic pool to audit.Source: C Detail Tables and Plots · p. 15

Benchmark composition

The benchmark comprises two classification datasets and four languages, a deliberately narrow scope that keeps counterfactual validation task-specific. Appendix tables extend the main comparisons with full downstream, audit, and per-cell views.

Benchmark composition

  • The completed replay covers two classification datasets and four languages, isolating classification evidence rather than mixing it with planned task extensions.Source: Benchmark composition · p. 15

Full per-selector downstream table (including gold-only)

  • Table 9 reports downstream Macro-F1 for all selectors and a gold-only reference, which is excluded from audit-versus-downstream correlations because it has no synthetic pool to audit.Source: Full per-selector downstream table… · p. 15

Full per-selector audit table

  • Table 10 presents the selected-pool audit view, including leakage, while the main text emphasizes LC, Q, hard-reject, C, and H.The passage says leakage means are small but selector differences remain visible.Source: Full per-selector audit table · p. 15

Per-cell Macro-F1: DEITA and CoSDA-EQ

Within-cell Spearman ranks

Within-cell judged-label-correctness and downstream Macro-F1 rankings show near-zero mean association and mixed signs. Appendix results reinforce that audit quality and downstream utility can rank selectors differently, with utility remaining cell-dependent.

Within-cell Spearman ranks

  • Near-zero mean association and mixed cell-level signs characterize the within-cell Spearman ranks between judged label correctness and Macro-F1 across selectors.Figure 1 presents this rank-correlation evidence.Source: Within-cell Spearman ranks · p. 15
  • Table 8 covers eight classification cells using Macro-F1; broader NER, intent/slot, and summarisation settings require task-specific validators.Source: Table 8 · p. 16
  • CoSDA-V2 sits near naive on mean Macro-F1, with their difference inside cell-level uncertainty according to the bootstrap confidence interval.Source: Table 9 · p. 16

Reading the appendix against the main paper

  • AlpaGasus has the largest downstream mean at 0.202, while CoSDA-V2 has the strongest judged label correctness at 0.904; utility remains cell-dependent.Source: Reading the appendix against the main… · p. 16
  • AlpaGasus’s D=0.590 is below every CoSDA variant, ruling out variety preservation as the source of its downstream lead; DEITA has the highest D.CoSDA-V2’s D is reported as approximately 0.60, bounded by available CoSDA variants at D∈[0.603, 0.604].Source: Table 10 · p. 17
  • DEITA wins 4 of 8 cells and CoSDA-EQ wins 1 of 8, while their aggregate means of 0.128 and 0.133 are both below naive.Source: Table 11 · p. 17

Related papers

Loading 2609.18960v1…