PaperlayerView exact revision on arXiv

Source-linked AI summary

Metric-Construction Coupling Inflates Measured Synthetic Dialect Recovery

Hyojung Han

arXiv:2610.00164v1cs.CL

TL;DR

Korean dialect corpora are licensed but cannot be redistributed for independent reproduction, leaving the recovery of real-data supervision by synthetic data and its independent measurement as open questions. The paper builds KoDialectBench and a recovery protocol, then tests construction-disjoint supervision and evaluator overlap. Recovery varies by axis, and marker-based generation estimates fall sharply when the scorer’s inventory is withheld from synthesis rules.

  • Problem

    The paper asks how much synthetic dialect supervision recovers under construction-independent evaluation and how metric–construction coupling biases that estimate.

    Source: §1 · p. 2

  • Method

    The paper introduces KoDialectBench and a matched-control recovery protocol, then tests construction-disjoint training and metric-construction coverage.

    Source: §1 · p. 2

  • Results

    Recovery is axis-dependent; in the matched 8,600-example disjoint comparison, dialectness recovery falls from 91.8% to 8.1% and region match from 101.9% to 25.6%.

    Source: §1 · p. 2, §6.3 · p. 10

  • Takeaways & Limitations

    The authors conclude that shared construction and evaluation inventories can substantially inflate estimates of synthetic-data recovery.

    Source: §1 · p. 2

    A stable external evaluator that is construction-independent and strongly discriminating is still missing.

    Source: §8 · p. 13

Abstract

from arXiv · show

Korean dialect corpora are available but not redistributable: weights may be released, while reproducing training and evaluation from the underlying data cannot be. We ask how much of that supervision synthetic data recovers, and whether that recovery can be measured independently of the synthesis pipeline. We contribute KoDialectBench, 1,000 items across five regions on three axes, released as identifier hashes and scoring code so users reconstruct the items from their own licensed copy. Recovery is strongly axis-dependent: our best synthetic arm reaches 91.2% of the real-data gain on region identification but 63.7% on comprehension. On generation the answer depends on the metric: the deployed marker lexicon reports 92.3% on dialectness and 119.3% on region match, the latter exceeding the real-data reference, whereas reference-based generation reaches 72.4%. We find the marker metrics' scoring inventory is entirely contained in the inventory our transformation rules can emit. We test the effect of construction access directly with an exact-form construction-disjoint arm that withholds 20% of marker types from the rules. At exactly matched training size (8,600 examples) it reduces dialectness recovery from 91.8% to 8.1% and region-match recovery from 101.9% to 25.6% on the held-out marker inventory, while the three pipeline-independent measurements do not fall at all. A complementary evaluator sweep defines metric-construction coverage (MCC) and finds measured dialectness recovery increasing monotonically as overlap rises from MCC=0 to MCC=1. Shared construction and evaluation inventories can therefore substantially inflate estimates of synthetic-data recovery.

1 Introduction

Korean dialect corpora are available, but licensing restricts redistribution and independent reproduction of training and evaluation. The paper introduces a benchmark and recovery protocol to measure synthetic supervision, finding that recovery varies by axis and generation metric independence matters.

1 Introduction

  • Rule-based synthesis is a common response to data scarcity, but the amount of real-data gain it recovers is seldom reported separately by ability under controlled conditions.The cited examples include VALUE and Multi-VALUE [50, 51] and DADA [27].Source: §1 · pp. 1–2
  • Recovery varies sharply by axis, and generation metrics can make synthetic supervision appear to surpass real data when they share the synthesis pipeline’s inventory.The paper treats that apparent surplus as a measurement failure and shows how to detect it.Source: §1 · p. 2
  • The paper asks how much synthetic dialect supervision recovers under construction-independent evaluation and how metric–construction coupling biases that estimate.A secondary diagnostic reports that marker-based over-separation does not survive a marker-free probe.Source: §1 · p. 2
  • KoDialectBench provides a three-axis Korean dialect benchmark through a licence-compatible reconstruction scheme, alongside a matched-control recovery protocol and metric-construction coverage (MCC).The contributions also include a construction-disjoint intervention that reduces marker-based recoveries while leaving pipeline-independent measurements within the observed seed difference.Source: §1 · p. 2

2 Related work

Related work spans rule-based dialect synthesis, synthetic-data evaluation, and benchmark validity. The paper distinguishes its focus—feature-inventory coupling between synthesis and scoring—from prior evaluation distortions and recovery comparisons.

2 Related work

  • Prior dialect synthesis includes VALUE [50], Multi-VALUE [51], DADA [27], and EnDive [14], while Korean work includes translation evaluation and dialect identification.The paper frames its setting as Korean regional dialects under a licence constraint and distinguishes its inventory-coupling failure from source-copying distortion [36].Source: §2 · pp. 2–3
  • Style-transfer evaluation work separates style strength, content preservation, and fluency [31], while reporting metric interdependence [46], weak task-dependent human correlation [5] [21], and inconsistent validation [34] [35] [44].The paper manipulates overlap between synthesis and scoring inventories rather than meta-evaluating metrics against human judgment; it reports no human evaluation.Source: §2 · p. 3
  • The paper relates its claim to construct validity and Goodhart-like proxy failure, but does not assign a specific Goodhart subtype because the marker metric is not the training objective.It also notes that test-data properties can inflate apparent metric validity.Source: §2 · p. 3
  • Prior synthetic-data work compares synthetic-trained models with real data [8], but the paper identifies a gap in expressing that comparison as recovery of a real-data arm’s gain over an untrained baseline.The authors state this as a gap, not a priority claim; recursive-training diversity loss [13] and tail degradation [1] concern a distinct phenomenon.Source: §2 · p. 3
  • Benchmark artifacts [15] [38], contamination [41] [42], and coupling differ: coupling shares the evaluation feature inventory with the training-data construction pipeline without leaking items or relying on annotator shortcuts.Synthetic evaluation sets can also be easier than human-written benchmarks, and automatic benchmarks can inflate scores without underlying behavior changing [17].Source: §2 · pp. 3–4

3 KoDialectBench

KoDialectBench contains 1,000 items across five Korean regions and three tasks, with separate marker-lexicon and reference-based generation scores. Its release design lets users reconstruct items from licensed data without distributing the text or marker lexicon.

3.1 Composition

  • The benchmark has 1,000 items—200 per region—and scores comprehension, five-way identification, and dialect generation using separate marker-lexicon and reference-based score families.The marker family includes dialectness, region match, and ADI2; the reference family uses chrF against the paired human dialect sentence.Source: §3.1 · p. 4

3.2 Establishing discriminative power

  • The marker-lexicon metric counts words carrying known regional markers, but it measures lexical dialect signal rather than naturalness or generation quality.On held-out pairs it separates dialect from paired standard sentences at 86.5–96.5% across regions, but that does not establish construct validity or pipeline independence.Source: §3.1 · p. 4, §3.2 · p. 4

3.3 Release under a restrictive licence

  • The licence permits release of trained weights but bars redistribution of corpus-derived material, including the token-label correspondence inventory.The artifact uses identifier hashes, body digests, isolated scoring code, and local lexicon regeneration so users reconstruct items from licensed copies; an eight-axis gate caught an interpreter-path dependency in an earlier check.Source: §3.3 · pp. 4–5

4 Experimental setup

The experiments use shared model and adapter settings across real, synthetic, and construction-disjoint arms, with controls for source register, seed, and training size. Before measuring recovery, the authors repaired baseline and adapter-serving issues and repeated generation under a fixed configuration.

4.1 Arms and training

  • All arms use the same bf16 base model and LoRA settings, with real and synthetic reference arms at 8,600 examples and an exactly size-matched construction-disjoint arm.Other arms include an independently sampled disjoint arm, a fixed-update control, a larger synthetic arm, and a seed repeat; training mixes tasks evenly and samples regions uniformly for identification.Source: §4.1 · p. 5

4.2 Two preconditions we had to repair first

  • Recovery was computed only after matching the base and reference model family and precision and verifying that merged adapters were active at serving.Because greedy decoding was not reproducible on the stack, cited numbers are medians of three repeats under one fixed serving configuration.Source: §4.2 · p. 5

5 The synthetic pipeline

The pipeline’s measured recovery depends on marker coverage, source-register applicability, and training-set size; improvements in rule applicability or quantity do not isolate downstream recovery from other constraints.

5 The synthetic pipeline

  • Three transformation refinements—frequency-preserving variant sampling, purity-based attestation tiers, and sampled density—produced no appreciable recovery change.The density was sampled from the observed real distribution rather than fixed.Source: §5 · pp. 5–6
  • Applicability was 22–62% of real rates (Jeolla 5.4% vs 23.2%); colloquial prompting raised it from 39.0% to 79.6%, while dialect-vocabulary steering did worse.Markers came from spoken conversation, but transformed sentences were written-register prose generated by a 27B model.Source: §5 · p. 6
  • At equal training size, colloquial sourcing changed comprehension by −0.49 chrF, identification by +0.60 pp, and reference-based generation by +0.43 chrF—only 0.3–0.5 times seed-to-seed variation.Rule coverage and downstream recovery are separate axes; an applicability gain should not be reported as a recovery gain.Source: §5 · p. 6
  • Increasing synthetic training examples from 8,600 to 26,600 improved region identification by 9.9 times the observed SK5–SK6 variation, while comprehension and reference-based generation stayed within it.Because every arm trained for one epoch, the larger arm also received roughly three times the optimizer updates, confounding data quantity and compute.Source: §5 · p. 6
  • 8.1% held-out dialectness recovery followed withholding one-fifth of marker types; raising transformation probability to p=1.0 cannot overcome the marker-inventory limit.The earlier per-region estimates were not quoted because they predated the structured ledger.Source: §5 · p. 6

6 Results

Across the results, recovery varies by capability axis, and generation estimates shift with metric–construction overlap. Restricting construction access sharply reduces held-out marker scores without lowering pipeline-independent measures, while the MCC sweep links greater inventory coverage to higher measured dialectness recovery.

6.1 Recovery differs by axis

  • SK4 recovers 63.7% on comprehension, 91.2% on identification, and 72.4% on reference chrF; deployed markers report 92.3% dialectness and 119.3% region match.The marker-based region-match estimate exceeds the real-data reference.Source: §6.1 · p. 7, Figure 1 · p. 7, Table 1 · p. 7

6.2 Metric–construction coupling

  • The scoring lexicon is contained in transform-emittable forms; independent reference chrF gives 72.4% generation recovery, versus 92.3% dialectness and 119.3% region match under coupled markers.No synthetic arm exceeds the real-data reference under the independent metric.Source: §6.2 · p. 8, Figure 2 · p. 8

6.3 Intervening at construction time

  • Construction-disjoint means exact-form disjoint, not scoring-disjoint: suffix matching still fired on 239 of 8,600 targets (2.78%), and this residual contamination favors higher held-out marker scores.The reported exact-form ban does not prevent every form from triggering the evaluator.Source: §6.3 · p. 9
  • At matched 8,600 examples, held-out dialectness recovery falls from 91.8% to 8.1% and region match from 101.9% to 25.6%, while independent measures do not fall.The comparison withholds 20% of marker types from the transformation inventory.Source: Table 3 · p. 9, §6.3 · p. 10, Figure 3 · p. 10
  • The intervention suggests construction access drives much of the marker-recovery difference, but shared source sentences prevent a fully isolated causal estimate.The primary matched arms share 66.2% of their source sentences.Source: Table 3 · p. 9, §6.3 · p. 10

6.4 Quantifying the coupling: MCC and an overlap sweep

  • Measured dialectness recovery rises from −54.8% at MCC=0 to 84.8% at MCC=1 while models and generations remain fixed.Only the evaluation inventory changes across the sweep.Source: §6.4 · pp. 11–12, Figure 4 · p. 12
  • As MCC falls, the real–base gap shrinks from 0.0449 to 0.0146 and recovery intervals widen from 23 to 201 points, though the evaluator never goes blind.The recovery ratio becomes less stable as its denominator shrinks.Source: §6.4 · p. 12, Figure 4 · p. 12

6.5 Meaning preservation is a guardrail

  • Meaning preservation is a guardrail, not a recovery axis; marker-based over-separation failed a marker-free probe and is excluded from quantitative claims.Source: §6.5 · p. 13

7 Seed sensitivity

The section uses one observed seed pair as an informal sensitivity reference, not as an estimate of training variance or a significance test.

7 Seed sensitivity

  • SK6’s observed seed-to-seed differences under SK5 were 1.83 chrF (T1), 1.1 pp (T2), 0.0009 dialectness, 0.0346 region match, and 4.9 pp reference-based generation recovery.The authors treat these as an informal sensitivity reference, not a variance estimate.Source: §7 · p. 13
  • SK4’s advantage over SK5 was large relative to the observed SK5–SK6 difference on identification, dialectness, and region match, but not on comprehension.For reference-based generation, the advantage was also within the observed difference.Source: §7 · p. 13
  • Increasing synthetic training data from 8,600 to 26,600 examples did not yield a comprehension improvement larger than the observed SK5–SK6 difference.The authors avoid calling this a plateau because other interventions were not tested.Source: §7 · p. 13

8 Limitations

The limitations concern evaluator independence and stability, incomplete isolation and replication of the disjoint comparison, and the scope of the metrics and experiments.

8 Limitations

  • No externally sourced evaluator is both construction-independent and strongly discriminating; at MCC = 0, the separation margin is roughly three times smaller than at MCC = 1.Recovery ratios from the decoupled evaluators are therefore substantially less stable.Source: §8 · p. 13
  • The primary disjoint comparison rests on one size-matched run, and its seed reference is partly borrowed from a different training scale.The available disjoint-condition seed pair is at 26,861 examples, not 8,600; a seed replicate of SK12-D9b would address this.Source: §8 · p. 13
  • SK12-D9b matches SK5 on size, task-by-region composition, source register, and optimization recipe, but shares only 66.2% of its source sentences.The authors characterize this as direct intervention evidence, not a fully isolated causal estimate.Source: §8 · p. 13
  • Over-separation is scoped to the marker axis, and the authors cannot tell whether the marker-free probe’s opposite direction reflects generations or the marker metric.Only one marker-free probe was available.Source: §8 · p. 14
  • The reported seed sensitivity is a single observed comparison at 8,600 examples under the coupled condition, extrapolated to larger and disjoint arms by assumption.It does not estimate training variance.Source: §8 · p. 14
  • Because every arm trains for one epoch, the 26,600-example arm also takes three times the optimizer steps, confounding data quantity with compute.A step-matched control would separate these factors.Source: §8 · p. 14
  • Both generation metric families have limitations: marker metrics are pipeline-coupled, reference metrics penalize legitimate variation, and neither is human judgment.The study did not run human evaluation; all results are on Qwen3.8-27B, with single-turn training and no conversational evaluation.Source: §8 · p. 14

9 Release

The release provides the benchmark and selected model artifacts while withholding recoverable training resources.

9 Release

  • KoDialectBench and checkpoints for the real-data reference arm R and synthetic arm SK4 are released.The primary intervention arms are also released as LoRA adapters with a merge script so Figure 3 can be rerun.Source: §9 · p. 14
  • Training sets, the marker lexicon, and the glossary are withheld because the dialect-to-standard triples can be recovered from the synthetic examples alone.The adapter must be merged and verifier-validated because an unmerged adapter is a silent no-op on this architecture.Source: §9 · p. 14

A All arms, all metrics

This block documents the all-arm metric table, control checks, and a marker-free regional profile.

A All arms, all metrics

  • Table 5 repeats Table 1 with arm C included, enabling direct inspection of the source-register comparison.It reports medians of three runs for every arm and benchmark metric; marker-family values use the deployed full lexicon, with held-out intervention results reported separately in Table 3.Source: A All arms, all metrics · p. 19, Table 5 · p. 20

B Control preconditions and their negative tests

  • The matched bf16 baseline exposed sign disagreements, while identical base and adapted log-probabilities to five decimals revealed a silent adapter failure.The merge script now checks for changed adapted tensors; the prompt templates also differ by one newline for T3, unlike T1 and T2.Source: B Control preconditions and their… · p. 20

C Marker-free region profile

  • The initial character n-gram divergence saturated at 0.963–0.977 across region pairs, so the authors discarded it.The paired edit-ratio profile avoids differing item-set content, while human references range from 0.134 to 0.306 by region.Source: C Marker-free region profile · p. 20

D A small out-of-domain probe

A small probe using six self-written scenes compared merged models across five regions, but its results are qualitative rather than a quantitative benchmark. Arm R ignored the requested region on a formal customer-service sentence, while the marker-free edit measure showed less movement and a narrower regional spread for the synthetic arm than for human references.

D A small out-of-domain probe

  • The probe used six self-written scenes across five regions, with no source text and one greedy run per model; Table 6 labels it qualitative, not a benchmark.The scenes covered customer service, a public notice, tourism, drama and game dialogue, and everyday speech; n = 30 per model.Source: D A small out-of-domain probe · p. 20, Table 6 · p. 20
  • Arm R returned the identical formal customer-service sentence for all five regions, while the probe’s three measures favored synthetic output on two and arm R on one.The third measure was model self-consistency, which reflects internal consistency rather than correctness; the authors draw no quantitative claim from this small probe.Source: D A small out-of-domain probe · pp. 20–21
  • On marker-free character edit ratio, all arms moved far less than human references, and the synthetic arm’s between-region spread was narrower rather than wider.The ratio is edit distance between each standard input and generated output, normalized by input length.Source: Figure 5 · p. 21

E A second apparent finding that we exclude

A marker-free probe did not confirm broader over-separation, and earlier exploratory marker-based figures were excluded because they could not be reproduced from released records. The checkpoint provenance check found no complete source-corpus utterance verbatim, while transformed words still largely reflected source-derived rules.

E A second apparent finding that we exclude

  • Earlier exploratory marker-based attribution figures suggesting sharper regional separation were excluded because they were not ledgerbacked or reproducible from released bench records and scoring code.Nothing in §§6.1–6.4 depends on the unresolved broader-distribution question.Source: E A second apparent finding that we… · p. 21
  • Arm R’s edit magnitude was 0.56 of human magnitude and SK4’s was 0.45; synthetic between-region spread was 0.31 of human against arm R’s 0.76.The marker-free probe is reported as a failure to confirm broader over-separation; whether the synthetic distribution is distorted more broadly remains unresolved.Source: E A second apparent finding that we… · p. 21

F Checkpoint provenance

  • Exact matching found 0 of 26,600 checkpoint examples and 0 of 8,600 SK12-D9b examples containing a complete source utterance, versus 8,600 of 8,600 for arm R.This check covered training sentences of at least six whitespace-delimited eojeols. It does not imply no source-derived material: 77.2% of transformed training words pass the mined rules, with 4.6% explained by public resources alone.Source: F Checkpoint provenance · pp. 21–22
Loading 2610.00164v1…