PaperlayerView exact revision on arXiv

Source-linked AI summary

A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfer

Wonsup Shin, Jingu Kim

arXiv:2608.20394v1cs.SDcs.CL

TL;DR

The paper addresses the lack of public stage-by-stage evidence for speech-to-SFT refinement pipelines. It independently toggles transcript and QA-quality refinement in a 2×2 study across Korean medical and finance data and nine LLMs. Quality improves consistently, but downstream MCQA gains are not uniform or significant on average, concentrating instead on family-domain aligned pairs.

  • Problem

    Public speech-to-SFT studies rarely isolate the marginal value of transcript and QA-quality refinement stages.

  • Method

    A 2×2 factorial independently toggles Phase 0 transcript refinement and Phase 2 quality refinement, evaluates four conditions across 40 Korean medical and finance sessions, and fine-tunes 9 LLMs.

  • Results

    QA quality improves by ∆2−0 = +0.18 with LLM judges and +0.22 with human experts, but the cross-cell mean MCQA effect is not significant and positive transfer concentrates on family↔domain aligned cells.

  • Takeaways & Limitations

    QA-data quality gains do not uniformly translate into MCQA gains under the fixed standard SFT recipe, consistent with a mismatch between explanatory items and factoid-recall evaluation.

  • Takeaways & Limitations

    The study does not apply multiple-comparison correction across all 72 individual cells, and family-varying effects are left for future work.

Abstract

from arXiv · show

Industry pipelines that turn speech into supervised fine-tuning (SFT) data via multi-stage refinement are increasingly adopted but, to our knowledge, have not been publicly ablated stage-by-stage, leaving each stage's marginal value unknown. We design a production-ready speech-to-SFT pipeline in which transcript refinement (Phase 0) and SFT data quality refinement (Phase 2) are independently toggleable, yielding a 2x2 factorial design. For each condition, we generate QA-form SFT data from Korean medical and finance conference recordings and fine-tune 9 models (5 LLM families, 2.4B-70B); we evaluate with four cross-provider LLM judges, a blind six-expert human evaluation, and 3 downstream MCQA benchmarks. Our central finding: under a fixed, standard SFT recipe, improvements in QA data quality do not transfer uniformly into downstream MCQA gains. 4-judge quality rises consistently, yet the cross-model mean MCQA gain is not significant; positive transfer concentrates on family-domain aligned pairs. This differential pattern is consistent with a format mismatch: Phase 2 shifts SFT-data composition toward explanatory items, while MCQA primarily probes factoid recall. All six human raters report higher full-pipeline quality, confirming the LLM-judge direction. An STT-engine swap to Whisper-medium confirms pipeline robustness. A non-hallucination audit shows the two frontier LLMs admit unknown on approximately 8% of QA on average; we release samples, prompts, code, and all SFT checkpoints.

1 Introduction

This paper addresses the lack of public per-stage ablations for speech-to-SFT refinement pipelines. It uses a factorial evaluation to test whether transcript and QA-quality refinement improve data quality and downstream transfer.

  • Public studies have not isolated the marginal value of transcript refinement and SFT-data quality refinement in speech-to-SFT pipelines.Prior work targeted general capabilities or continuous pre-training rather than per-stage ablation in a domain-specific setting.
  • A 2×2 factorial independently toggles Phase 0 transcript refinement and Phase 2 quality refinement while sharing Phase 1 QA generation.The study uses Korean medical and finance conference content across 40 sessions.
  • Nine LLMs spanning five families and 2.4B–70B parameters are fine-tuned on each condition and evaluated with LLM judges, human experts, and downstream MCQA benchmarks.The evaluation includes four LLM judges, six expert raters, and KMMLU, KMMLU-Pro, and MMLU.
  • QA-quality improvements do not transfer uniformly to downstream MCQA under a fixed standard SFT recipe.Positive transfer concentrates on family↔domain aligned pairs, while the cross-model mean MCQA gain is not significant.
  • LLM judges and human experts both confirm robust QA-quality improvement, while the pattern is consistent with a mismatch between explanatory SFT data and factoid-recall MCQA.The reported gains are +0.18 for LLM judges and +0.22 for human evaluation on a 1–5 scale.

2 Related Work

Related work includes synthetic document-to-SFT and speech-to-SFT datasets, but these studies do not provide a per-stage ablation of speech-to-SFT refinement. The paper positions its domain-specific factorial design as addressing that gap.

  • Synthetic SFT methods such as Self-Instruct, Evol-Instruct, Cosmopedia, EntiGraph, and GLAN generally construct data from seeds, web text, or clean documents.They do not address speech-specific transcription noise, named-entity grounding, or discourse structure.
  • Recent speech-to-SFT work includes COSMIC, SIFT-50M, LiveCC, and PodGPT, but these target general capabilities or use continuous pre-training rather than text-LLM SFT.The cited systems address different speech-data construction or training settings.
  • None of the cited speech-to-SFT studies performs a per-stage ablation of refinement stages.The paper addresses this gap with a 2×2 factorial ablation on Korean medical and finance content.
  • The study evaluates QA quality with four cross-provider LLM judges and uses Korean and English MCQA benchmarks for downstream assessment.KMMLU covers Korean subjects, KMMLU-Pro adds professional-license-exam items, and MMLU provides an English cross-lingual sanity benchmark.

3 Pipeline and Dataset

The pipeline separates transcript refinement, QA generation, and QA-quality refinement, with Phase 0 and Phase 2 independently toggled. It is applied to consented Korean medical and finance conference sessions and produces condition-specific QA datasets.

  • Pipeline architecture: Phase 0 cleans segmented STT output using discourse summaries, multi-STT cross-validation, NER-based retrieval augmentation, and unified proofreading.Multi-STT validation re-invokes secondary providers on source audio when Phase 0 is active.
  • Pipeline architecture: Phase 1 converts transcripts into QA through analysis, context augmentation, strategy, and generation, retaining domain, sub-domain, difficulty, and question-type metadata.Question types include factoid, explanatory, procedural, and comparative items.
  • Pipeline architecture: Phase 2 applies a five-judge filter, up to two refinement rounds, and embedding-based deduplication before discarding items that still fail.Its verbatim prompts are released because Phase 2 is the ablation’s measurement instrument.
  • Factorial design: The 2×2 design independently toggles Phase 0 and Phase 2 while keeping Phase 1 shared.This yields four factorial conditions for comparing transcript and quality refinement.
  • Dataset: Forty consented conference sessions comprise 19 medical and 21 finance sessions, with primarily Korean audio and multilingual segments preserved.Per-condition QA counts range from 2,598 to 2,765, a roughly 6% spread.
  • Evaluation sample: A stratified 200-QA sample supports LLM-judge, human-quality, and difficulty evaluations, with a Phase-2-filtered parallel sample for sensitivity analysis.The sample is drawn from training data but does not query the trained SFT models.

4 Experimental Setup

The experiments span 72 main LoRA cells across nine models, four factorial conditions, and two domains, using multiple evaluation axes and factorial effect decompositions. Seed replication and bootstrap or mixed-effects analyses quantify uncertainty.

  • Models and runs: The main experiment contains 72 LoRA cells spanning 9 LLMs, 4 conditions, and 2 domains across 5 model families and 2.4B–70B sizes.Additional experiments include 12 Full FT sanity runs and an 8-cell Whisper-medium STT-swap grid.
  • Training: Main LoRA cells use two seeds, while Full FT and auxiliary experiments use specified alternative seed schemes.The training setup includes QLoRA 4-bit quantization, rank r = 16, scaling α = 32, dropout 0.05, and three epochs.
  • Evaluation: Evaluation covers cell-blind four-judge QA scoring, blind six-expert QA evaluation, downstream MCQA, Whisper-medium robustness, and an LLM-difficulty audit.The judges score a 200-QA sample using a 1–5 Likert rubric.
  • Statistics: The full-pipeline effect is defined as ∆2−0 ≡ Exp 2−Exp 0, with separate single-stage and marginal factorial decompositions.The QA-quality decomposition exposes interaction, whereas the MCQA decomposition averages over the other factor.
  • Statistics: Bootstrap confidence intervals use 100,000 resamples, and mixed-effects regression separates between-cell and seed-level variance.The reported analyses complement the factorial effect estimates.

5 Results

The full pipeline consistently improves QA quality, but downstream MCQA effects are conditional rather than uniform across models and domains. STT substitution is broadly robust, while unknown-admission and full-fine-tuning checks provide additional validation.

  • QA data quality: +0.18 4-judge mean QA quality gain, from 3.81 to 3.99 on the 5-point scale.The frontier subset gains +0.23, and all five evaluated dimensions improve from Exp 0 to Exp 2.
  • QA data quality: Phase 2 shifts composition toward explanatory questions, but within-type improvements account for approximately 86% of the full QA-quality gain.Reweighting Exp 2 to Exp 0 composition yields type-controlled ∆2−0 ≈+0.16, while composition shift contributes approximately 14%.
  • QA data quality: +0.22 pooled human QA-quality gain, with all six raters reporting positive full-pipeline effects.Medical inter-rater agreement reaches ICC(2,3)=0.69 and Krippendorff α=0.42.
  • Factorial main effects on MCQA: All six MCQA aggregates show positive full-pipeline effects from +0.04 to +0.50 pp, but signs remain model-family-conditional.The Phase 0 × Phase 2 interaction varies in sign and magnitude across aggregates.
  • Factorial main effects on MCQA: Positive domain-aligned MCQA cells include EXAONE 3.5 7.8B on finance (+1.98 pp), Qwen 3.5 9B on finance (+1.71 pp), and Phi-4 Mini on medical (+2.10 pp).Negative cells cluster on both Gemma 3 sizes, including −1.46 pp on medical and −2.11 pp on finance for Gemma 3 27B.
  • Factorial main effects on MCQA: The cross-cell mean MCQA effect is +0.31 pp with 95% CI [−0.24, +0.86] and paired t p=0.28.A mixed-effects analysis attributes 99.4% of total variance to between-cell model×domain heterogeneity rather than seed noise.
  • W-grid: Whisper-medium reproduction: Whisper-medium changes 4-judge QA quality by at most 0.21 in absolute value across four cells.The across-cell 4-judge mean difference is −0.02.
  • W-grid: Whisper-medium reproduction: Whisper-medium’s mean absolute MCQA deviation is ≤1.32 pp on KMMLU and ≤0.39 pp on MMLU at both pipeline endpoints.Aggregate signed-bias reductions are approximately 26% on KMMLU and 60% on MMLU, driven by sign cancellation rather than per-cell improvement.

6 Discussion

The pipeline reliably improves QA-data quality across LLM judges and human experts, but those gains do not transfer uniformly to downstream MCQA. Practical benefits depend on family–domain alignment and the task format, while STT substitution is robust only at aggregate level.

  • Quality and transfer: Per-judge quality gains range from +0.08 to +0.24, and six human raters likewise support the full-pipeline improvement.This cross-instrument convergence strengthens confidence beyond single-instrument evaluation bias.
  • Quality and transfer: QA-quality improvement does not yield a significant cross-cell mean MCQA gain, with positive transfer concentrated on family–domain aligned cells.A single LoRA recipe across all 72 cells may underfit some model families, making family-specific tuning a plausible contributor.
  • Data hardness: The two frontier LLMs admit unknown on 7.8% of QA, reflecting the session-grounded, institution-specific character of the source audio.The paper presents this bound as a low-cost triage tool for assessing new corpora and their differentiation from open-web augmentation.
  • Industry implications: Cross-cell mean KMMLU transfer is not significant, while Multi-STT Phase 0 substitutes proprietary STT at the aggregate level.The authors recommend per-cell validation before scaling to new model families or fixed deployments; Phase 2 best fits QA-format-aligned tasks.

Limitations

The study’s limitations concern evaluation scope, statistical modeling, sampling, and pipeline coverage. Results are therefore strongest for the tested Korean domains, fixed SFT recipe, selected endpoints, and sampled audit data.

  • Sampling and measurement: Quality and difficulty analyses use a stratified 200-QA public sample rather than the full training set, leaving some sample-bias risk.The sample is drawn from training data but does not evaluate the SFT models themselves.
  • Sampling and measurement: Per-judge effects span +0.08 to +0.24, and three of four judges favor Phase 0 in the single-stage decomposition.This indicates cross-judge variation in the quality comparison.
  • Statistical scope: The study does not correct for multiple comparisons across all 72 cells, and family-varying effects are not formally modeled.The reported average domain-aligned effect is not distinguishable from zero, while family-level random slopes or interactions remain future work.
  • Pipeline and evaluation scope: The 72-cell ablation uses a fixed LoRA recipe, leaving transfer sensitivity to longer training, alternative objectives, larger data, and curricula untested.The 12-run Full FT sanity check is the only deviation from the standard recipe.
  • Pipeline and evaluation scope: MCQA is the downstream endpoint, so open-ended QA and RAG-QA could provide complementary format-aligned evaluation.Broader endpoint coverage is explicitly left for future work.
  • Pipeline and evaluation scope: Phase 0 is evaluated as one block, preventing attribution to its summary, multi-STT, NER-based RAG, and proofreading sub-steps.Isolating these internal components remains future work.
  • Pipeline and evaluation scope: Only Whisper-medium is tested for STT robustness, while broader production engines and model-tier coverage remain future work.The W-grid focuses on EXAONE 3.5 7.8B and Qwen 3.5 4B across medical and finance.
  • Statistical scope: Condition-level QA counts vary by up to 167 items, reflecting phase-selective retention rather than an equalized comparison.Equal-count subsampling is described as potentially biasing the comparison by removing selectively retained items.

Ethics Statement

The study uses consented Korean medical and finance conference data with anonymized released samples and externally recruited blind evaluators. Reproducibility artifacts are released within research and non-commercial-use constraints.

  • Data and consent: The 40 sessions come from consented Korean-language service data spanning medical and finance conferences, seminars, and industry forums.Speakers consented to research and non-commercial use, but source audio, full transcripts, and the complete generated corpus are not released.
  • Privacy: Released QA samples replace personally identifiable information with neutral placeholders while retaining generic organization names as domain anchors.Those retained names are not authors’ affiliations.
  • Human evaluation: Six blind expert evaluators were recruited externally, had no employment or design relationship with the authors, and were compensated.Only anonymized professional profiles are disclosed.
  • Artifacts and constraints: The repository releases pipeline identifiers, scripts, configurations, rubrics, prompts, and the 200-QA samples; 172 SFT checkpoints are also released.Downstream use must respect the research and non-commercial consent scope.
  • Ethics and evaluation: The work follows the ACL Code of Ethics and limits human-subject involvement to consented data and voluntary, anonymized blind evaluation.The evaluator profiles disclose only role, sub-domain, and experience range.
  • Resource reporting: Per-QA token-cost estimates are reported for Phase 0 at 0.011–0.014 USD and Phase 1 at 0.040–0.050 USD.The estimates use OpenRouter unified pricing from a representative log sub-sample.

F Full FT sanity per-cell results

The Full FT sanity analysis compares stage-wise deltas with LoRA results across selected model-domain cells. Absolute learning effects show strong agreement between the two fine-tuning methods.

  • F Full FT sanity per-cell results: Stage-wise deltas are reported on training-domain-aligned KMMLU and MMLU subsets across 12 cells.The comparison covers 3 models, 2 domains, and 2 benchmarks; the LoRA comparison uses the seed-42 run.
  • F Full FT sanity per-cell results: 11/12 cells agree on the sign of Exp 2−base learning effects between LoRA and Full FT.The correlation is r = 0.841, with a median gap of 0.81 pp.
  • F Full FT sanity per-cell results: The lower Exp 2−Exp 0 agreement reflects small-model LoRA instability rather than systematic divergence at medium scale.
  • F Full FT sanity per-cell results: The companion repository provides per-cell base accuracies used to derive the absolute Exp 2−base learning effects.

I KMMLU-Pro per-cell results

KMMLU-Pro evaluation shows mixed per-cell effects of Phase 2 across medical and finance domains. The benchmark covers all nine models with license-aligned domain subsets.

  • I KMMLU-Pro per-cell results: All 72 SFT cells across 144 seeded runs are complete, with aggregates computed over all nine models.Medical-trained cells use 1,205 KMMLU-Pro medical items, and finance-trained cells use 801 finance items after license-aligned filtering.
  • I KMMLU-Pro per-cell results: Medical ∆2−0 ranges from −1.57 pp to +2.12 pp, while finance ∆2−0 ranges from −1.19 pp to +1.31 pp.The medical endpoints are Qwen 3.5 9B and Gemma 3 4B; the finance endpoints are Llama 3.3 70B and EXAONE 3.5 7.8B.
  • I KMMLU-Pro per-cell results: Medical signs are mixed at 5 positive and 4 negative, while finance has 5 positive, 3 negative, and 1 zero cell.
  • I KMMLU-Pro per-cell results: EXAONE 3.5 7.8B finance is the strongest positive cell on both KMMLU and KMMLU-Pro.Its gains are +1.98 pp on KMMLU-aligned evaluation and +1.31 pp on KMMLU-Pro.
  • I KMMLU-Pro per-cell results: Four cross-provider judges provide QA-quality scores alongside the downstream KMMLU-Pro evaluation.

L W-grid per-cell results

The Whisper-medium W-grid tests whether replacing the in-house STT engine changes downstream per-cell results. Effects vary by cell, while the filtered-sample quality effect remains close to the primary estimate.

  • L W-grid per-cell results: Qwen 3.5 4B finance shifts from −0.09 to +1.14 pp when comparing Whisper-medium with in-house STT.This sign flip drives aggregate cancellation in the W-grid comparison.
  • L W-grid per-cell results: The W-grid evaluates eight cells across EXAONE 3.5 7.8B and Qwen 3.5 4B in medical and finance conditions.The design compares W-Exp0 and W-Exp2 against matched in-house-STT cells.
  • L W-grid per-cell results: Only EXAONE 3.5 7.8B finance reduces its absolute STT difference, from 0.29 to 0.14 pp.The other three evaluated cells move farther from zero.
  • L W-grid per-cell results: The Phase-2-filtered sample yields a 4-judge ∆2−0 of +0.19, within 0.01 of the primary sample’s +0.18.The cross-judge sign pattern is preserved, supporting robustness to the sampling design.

S Human evaluator profile and detailed results

Human experts, judge analyses, and supplementary tables provide detailed quality, robustness, and downstream results. Across the reported quality evaluations, the full pipeline consistently outperforms the baseline condition.

  • S Human evaluator profile and detailed results: All six external human raters report positive ∆2−0 on the five-dimension QA-quality rubric.The blind evaluation uses 25 QA items per condition per rater, with three medical and three finance experts.
  • S Human evaluator profile and detailed results: Phase 2 shifts explanatory questions from 30% to 40%, while Phase 0 alone increases factoid concentration.Within-type quality improvement accounts for approximately 86% of the Phase-2 effect, versus approximately 14% from composition shift.
  • S Human evaluator profile and detailed results: The Phase-2-filtered sample reproduces the primary quality result with ∆2−0 = +0.19 versus +0.18.The preserved per-judge sign pattern indicates the effect is not specific to the primary sample selection.
  • S Human evaluator profile and detailed results: The keyword heuristic reports unknown-admission rates of 15.8% for medical and 21.0% for finance.Its fixed eight-keyword list is recall-limited and is reported as a supplementary rather than primary measure.
  • S Human evaluator profile and detailed results: KoBEST remains stable across conditions, with an approximately 2 pp drop only for Llama 3.3 70B versus its published base.The table describes this as suggesting mild forgetting at the Large tier.
  • S Human evaluator profile and detailed results: All five QA-quality dimensions show positive Exp 2−Exp 0 deltas in both LLM-judge and human evaluations.
Loading 2608.20394v1…