PaperlayerView exact revision on arXiv

Source-linked AI summary

No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus

Theodore O. Cochran, Stephanie Dodson, Keith Nore

arXiv:2608.28875v1cs.CLcs.AIcs.SD

TL;DR

This paper tests whether inference-time prompt context adapts production oral-history transcription, using a preregistered paired ablation across deployed configurations. Full context produced no detectable aggregate WER improvement, while rerun variability limited resolution of effects at that scale.

  • Problem

    Inference-time domain context is an inexpensive ASR adaptation mechanism, but evidence for its effectiveness in production oral-history transcription is limited.

  • Method

    The study preregistered a within-item paired ablation that reprocessed 19 cassette sides through the production path under three prompt arms and two deployed configurations.

  • Results

    Full prompt-level context produced no detectable change in aggregate WER, while gpt-4o-transcribe’s observed median paired difference was +0.6 WER points.

  • Takeaways & Limitations

    Context effects were small at side level but included improved coverage of listed phrases, supporting sequence-aligned term-level and speaker-label measures alongside aggregate accuracy.

  • Takeaways & Limitations

    Run-to-run pipeline variability exceeded observed arm differences, so effects of that magnitude cannot be reliably resolved from one transcription per cell.

Abstract

from arXiv · show

Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.

1 Introduction

The study tests whether prompt-level context improves transcription in a production oral-history tool, using an audited preregistered paired ablation. It finds no detectable aggregate WER change, while exploratory analyses suggest configuration choice and context-specific measures matter more than prompt engineering.

  • Motivation: Prompt conditioning supplies names, biographies, and vocabulary at inference time, motivated by earlier reports of substantial WER reductions on specialized audio.Production transcription tools increasingly expose this mechanism as a first-class feature.
  • System and question: The study examines Dialog Scribe, a production oral-history transcription tool using rich project-level context on degraded 1970s–80s cassette interviews.The corpus comes from a private Skagway, Alaska oral-history collection with operator-corrected verbatim references.
  • Confirmatory findings: No detectable aggregate accuracy change emerged: none of the four registered hypotheses was supported for either deployed model.For gpt-4o-transcribe, the observed single-pass side-level differences concentrated near zero, whereas Gemini estimates were unstable.
  • Operational comparison: A median 17.8-point WER gap between deployed configurations exceeded the prompt-arm differences, and pathological outputs occurred more often for the weaker configuration.The weaker configuration exceeded 100% WER on 5 of 19 no-prompt sides, versus 1 of 19 for the stronger configuration.
  • Interpretation: Context improved sequence-aligned performance on complete listed phrases, but the effects were too small or unstable to materially alter side-level WER.For Gemini, the listed-phrase improvement coexisted with worsened unlisted-token error, motivating term-level and speaker-label measures.
  • Implications: The paper recommends preregistration, frozen analysis code, implementation audits, and construct-specific evaluation when interpreting null ASR results.The proposed workflow treats null findings as requiring direct checks of manipulation validity and measurement validity.

2 Background and Related Work

Prior work presents prompt conditioning as a productized but unevenly supported adaptation mechanism. The paper situates its contribution as a preregistered test of the exact shipped mechanism and connects it to more mature decode-time biasing and construct-specific evaluation.

  • Prompt conditioning: Free-text conditioning is widely productized, but evidence for plain inference-time prompting remains recent, thin, and uneven.Earlier apparent gains often involve task engineering, exemplars, dedicated prompt encoders, or fine-tuning rather than plain text prompts.
  • Prior evidence: Direct evaluations report mixed results, including little or no average WER change and strong dependence on the model and evaluation setting.Related benchmarks identify a context-utilization gap and model-dependent exploitation of named-entity context.
  • Contribution: This study adds a preregistered, implementation-audited evaluation of the exact shipped mechanism on a production-derived corpus through the production code path.The contribution targets deployment validity rather than a reconstruction of the application’s behavior.
  • Alternative mechanisms: Decode-time biasing forms a more mature family, using structured bounded lists through WFSTs, shallow fusion, phrase-list attention, and related mechanisms.These approaches condition decoding rather than relying solely on an instruction-channel prompt.
  • Evaluation: WER can misrepresent downstream utility, motivating semantic, entity-aware, custom-vocabulary, speaker-attribution, and hallucination-specific measures.The paper’s results are framed as an example where context effects are excluded or diluted by aggregate WER.
  • Research practice: Preregistration separates confirmatory from exploratory claims and supports treating null results as publishable evidence rather than file-drawer residue.The paper follows this practice with frozen hypotheses, hashed analysis code, disclosed pilot exposure, and a deviations policy.

3 System Under Study

Dialog Scribe is a production web application for oral-history transcription in which operators attach names, biographies, vocabulary, and domain templates to uploaded audio projects. The study draws on a single operator’s degraded cassette corpus and uses corrected references as the scoring standard.

  • Application: Dialog Scribe lets operators attach interviewer and interviewee names, biographies, curated vocabulary, and a domain template to each uploaded audio project.The application renders these fields into a prompt at transcription time and sends it through provider-specific paths.
  • Production corpus: The production database contains 251 uploaded audio parts, approximately 40 hours of recordings, and 331 transcriptions from one operator’s sustained workload.The study’s retained sample is drawn from this production environment rather than a purpose-built benchmark.
  • Corpus characteristics: The recordings are degraded 1970s–80s cassette interviews from one private Skagway oral-history collection involving one interviewer and eight interviewees.Context was attached to nearly every project, so the observational corpus lacked a natural context-off comparison.
  • References: Operator-corrected verbatim references define the scoring standard, but extensive corrections mean “gold” denotes a reference standard rather than error-free ground truth.Reference-versus-machine-draft WER ranged from 15–50%.

4 Method

The method is a retrospective paired within-item ablation crossing three context arms with two deployed models on 19 cassette sides. Preregistered outcomes were computed from identically normalized side-level transcripts using paired statistical tests, with explicit implementation and validity safeguards.

  • Design: The experiment reprocessed the same audio sides under three context arms and two models, holding each model fixed within every paired contrast.The arms were none, structure, and full; the models were gpt-4o-transcribe and gemini-2.5-flash.
  • Design limitations: The within-side paired difference controls shared item composition, reference conventions, and recording difficulty, but the retrospective design does not provide randomized execution order or immunity to provider drift.Arm order was fixed rather than counterbalanced, and the sample followed ordinary reference availability.
  • Sample: 19 cassette sides formed the confirmatory sample, representing approximately 10.6 hours and 70 audio parts after two registered structural exclusions.Each side used the same reference in every experimental cell.
  • Prompt arms: The none arm used no prompt, structure supplied generic domain and speaker instructions, and full supplied the production template, named-speaker instructions, biography, and curated vocabulary.The full prompts were 912–1245 characters per project, while structure prompts were 417 characters.
  • Provider paths: OpenAI used size-split sub-chunks while Gemini used whole-file calls, producing 1,290 physical provider calls for 420 part-level records.OpenAI repeated prompts across sub-chunks and Gemini placed context inside a fixed provider wrapper.
  • Outcome measures: WER, CER, entity recall, and over-biasing were computed after identical normalization of all arms and references at side level.Entity recall used aligned gold proper-noun tokens, while the registered over-biasing measure tracked injected-vocabulary tokens.
  • Statistical plan: The preregistered analysis tested four directional hypotheses with two-sided Wilcoxon signed-rank tests on per-side paired differences, using Holm correction and bootstrap intervals for median effects.The primary contrasts used n = 19 sides per model, with a sensitivity analysis excluding two disclosed pilot sides.
  • Preregistration: The analysis plan and code were registered before the full dataset was scored, and deviations were reported as none.Post-freeze additions were labeled post hoc rather than treated as changes to the frozen scripts.

5 Results

Prompt-level context did not detectably improve aggregate transcription accuracy, while sequence-aligned analyses revealed targeted gains on listed phrases that were too small to materially change side-level WER. The two deployed configurations also differed substantially in accuracy and reliability, and Gemini’s context effects were unstable.

  • Confirmatory results: None of the four registered hypotheses was supported for either transcription configuration.The preregistered tests covered WER, entity recall, full-versus-structure context, and over-biasing.
  • Confirmatory results: +0.6 WER points was gpt-4o-transcribe’s median full-minus-no-context difference, with a 95% interval of [−1.1, +1.0].The interval comes from registered side-level resampling of observed single-pass outputs; it was not preregistered as an equivalence bound.
  • Configuration comparison: 17.8 WER points was the median no-prompt gap favoring gpt-4o-transcribe over gemini-2.5-flash on the same audio.This compares deployed provider pipelines, which also differed in segmentation, instruction wrapper, and endpoint type, not model identity alone.
  • Listed-term coverage and alignment: 1.7 percentage points was gpt-4o-transcribe’s listed-token error reduction under full context, while unlisted-token error stayed flat.Because listed tokens represented 4.6% of corpus reference positions, the corresponding component was approximately 0.08 WER points.
  • Listed-term coverage and alignment: Gemini’s complete-phrase gains coexisted with sharply worsened unlisted-token error and arm-unstable pathological outputs.Its larger count-based coverage gain therefore reflected a prompt-sensitive lexical effect entangled with output instability rather than a clean accuracy improvement.
  • Listed-term coverage and alignment: −6.0 percentage points was the corpus-weighted complete-listed-phrase error change for gpt-4o-transcribe, occurring on 13 of 19 sides in the distribution check.The change remained between −7.7 and −4.5 percentage points under leave-one-interviewee-out deletion and was −6.6 points after excluding the three most frequent terms.

5.5 Labelled post-hoc addition: endpoint-stochasticity rerun (2026-07-18)

A post-hoc five-repetition rerun showed that endpoint stochasticity exceeded the observed prompt effects, especially in unstable Gemini cells. The findings constrain interpretation of the single-pass confirmatory null rather than establish repeated-run equivalence.

  • gpt-4o variability: gpt-4o rerun variability ranged from stable moderate-side behavior to occasional large cell-level draws.Rep-to-rep WER standard deviations were 0.04–0.1 points on the moderate side and 0.5–5 points elsewhere; one structure repetition reached 77% WER against a 42% median.
  • Gemini variability: Gemini’s two unstable sides showed extreme none-arm variability, including standard deviations of 167 and 80 WER points.The rerun included single draws from 39% to 525% WER, indicating heavy-tailed behavior on unstable sides.
  • Interpretation: Run-to-run variability exceeded the observed gpt-4o prompt difference, so one transcription per cell could not reliably resolve effects of that size.Across 20 rerun pairs, the median paired ∆WER was +0.2 points, with IQR [−1.9, +2.7] and range [−10.2, +25.0].
  • Pathology: Pathological output was a stochastic, arm-dependent rate rather than a stable property of a side.One Gemini-only side had a 283% main-run draw but a 41% rerun median; across the rerun, full context produced zero pathological draws on that side.
  • Arm-dependent instability: Full context reduced some Gemini pathological draws and dispersion, but median WER rose on S06.On S03, MAD fell from 140.8 to 3.0 and pathological draws from 5/5 to 2/5; on S06, the maximum fell from 222.9 to 71.6 while median WER rose from 41.3 to 55.1.
  • Sequence-aligned measure: The rerun did not test whether the corpus-level sequence-aligned effect would reproduce, because the four-side subset carried little of that effect.The subset’s gpt-4o listed-token difference was −0.3 percentage points in the main run and remained between −0.3 and +0.2 across repetitions, versus −1.7 points corpus-wide.
  • Limitations: The rerun was descriptive and limited by post-hoc side selection, a seven-day interval, and unversioned provider aliases.These factors confound time with draw, and the four-side rerun cannot estimate resolution across the complete 19-side corpus.

6 The Implementation Audit

The implementation audit found that the production context manipulation was constructed, delivered, and consumed by both endpoints. Context changed speaker-label and listed-term production, but those effects were too sparse or unreliable to materially alter side-level WER.

  • Arm construction: The audit verified that prompt arms followed the application’s production code path and differed in prompt-level context.The off_generic arm used prompt=None, off_structure used a 417-character generic template, and the full arm used the production history render of 912–1245 characters.
  • Context contents: Full prompts contained participant names, a biography line, and a 34-entry vocabulary of local and domain-specific terms.The context was attached per project and rendered into the prompt used for transcription.
  • Delivery verification: OpenAI received the prompt on every call and Gemini received it inside a fixed wrapper, while logged prompt lengths matched all expected renders.The audit found zero mismatches across 420 records and no full-arm prompt missing its vocabulary.
  • Observed reactivity: Speaker-name labels and rare listed terms appeared more often in full-context outputs, demonstrating prompt-sensitive output changes rather than correctness.Labels were not scored against speaker turns, and Gemini’s unusually high counts were consistent with over-segmentation or repetition.
  • Mechanistic effects: Context increased listed-term production while speaker-label generation was excluded from WER by normalization.The audit reports +7.8 to +14.6 percentage points in bag-of-phrases occurrence coverage, with sequence-aligned gains concentrated on complete listed phrases.
  • Why aggregate WER stayed unchanged: The listed-term effects were worth under 0.1 WER points because relevant listed tokens were sparse among approximately 110,000 gold tokens.The manipulation worked, but its clearest lexical effects were too limited to materially change side-level WER on this corpus.
  • Design nuance: The registered H3 contrast includes the history-versus-general template difference, not a surgical isolation of individual context components.Separating template, names, biography, and vocabulary was identified as future work.

7 Discussion

The discussion argues that prompt-level context changed domain-specific outputs without changing aggregate side-level accuracy, while configuration choice mattered more operationally. It therefore recommends measuring contextual effects with sequence-aligned constructs rather than WER alone.

  • Operational implications: The two transcription configurations differed much more in observed WER than the prompt arms did.The configuration gap was exploratory, and the pipelines were not matched.
  • Future mechanism: Decode-time contextual biasing is presented as a stronger or more direct next candidate for domain-salient term gains, without condemning context generally.The present null concerns the shipped prompt channel specifically.
  • Construct interpretation: Context increased listed-term coverage and produced named speaker labels, but these changes were diluted or excluded by aggregate WER.Speaker labels were stripped during normalization, while the listed-term sequence-aligned gain was under 0.1 WER points against about 110,000 gold tokens.
  • Construct interpretation: Evaluations of contextual mechanisms should make sequence-aligned listed-term error and attribution accuracy first-class outcomes alongside aggregate accuracy.Related constructs measured on the same output can move independently.
  • Relation to prior work: The earlier ATC prompting result and this null are reconciled only by a hypothesis that prompt-addressable vocabulary differed across corpora; the mechanism was not demonstrated.The earlier study reported prompts roughly halving WER on Whisper Small and Medium.

8 Threats to Validity

The validity analysis treats the null as requiring evidence that a real effect was not simply missed. Internal threats were substantially mitigated, but inference remains bounded by single-pass stochasticity, clustering, construct choices, reference anchoring, and mechanism scope.

  • Internal validity: The within-item design removed arm-composition differences, while the audit strongly disfavored wholesale prompt-manipulation failure.All sides contributed all three arms, and 420/420 logged prompt lengths matched expected renders.
  • Internal validity: Reference anchoring favored the full-context arm’s wording and spelling conventions, making a context benefit somewhat easier to observe despite extensive corrections.Gold-versus-draft WER ranged from 15–50%.
  • Statistical-conclusion validity: For gpt-4o-transcribe, the 95% bootstrap CI for median paired ΔWER was [−1.1, +1.0] points, but it describes observed single-pass outputs rather than repeated-run expectations.No equivalence margin or smallest effect size of interest was preregistered.
  • External validity: The nominal n = 19 per model overstates independent information because sides cluster within eight interviewees, including one family contributing 9 of 19 sides.The safest reading concerns the observed corpus rather than an underlying speaker population.
  • Statistical-conclusion validity: One transcription per cell leaves endpoint stochasticity unmodeled, and the rerun found run-to-run variability exceeding the confirmatory differences.A TOST-style equivalence design is identified as the appropriate follow-up upgrade.
  • Construct validity: Aggregate WER can miss domain-salient token effects, while the entity-recall construct dilutes effects on the approximately 34 prompted terms.The paper bounds its conclusion to aggregate accuracy and does not claim that context lacks domain value.
  • Construct validity: The entity-recall metric is ad hoc and its aligned listed-term results depend on membership rules, which can reverse Gemini’s apparent direction.The paper recommends registering aligned listed-term error as a first-class outcome and varying membership rules where feasible.
  • Construct validity: The reference encodes one transcriptionist’s conventions, so absolute WER conflates ASR error with editorial choices; paired differences cancel the shared offset.Speaker labels were excluded from WER by design, and label correctness was not scored.

9 Limitations and Future Work

Future work should replace the study’s bounded single-corpus evidence with preregistered equivalence testing and independent references across additional archival collections. It should also evaluate contextual effects with sequence-aligned measures.

  • Future work: A follow-up should use a TOST equivalence test with a prespecified smallest effect size of interest.This would convert the current sensitivity bound into a registered claim.
  • Future work: A model bake-off on a second governed archival collection should use independently produced references to remove reference anchoring.The proposed follow-up also includes sequence-aligned evaluation.

10 Ethics and Data Governance

The study used a privately controlled historical oral-history collection with owner authorization, commercial API processing, and redacted, non-released data. It reports no IRB review and acknowledges residual reidentification risk from contextual examples.

  • Collection and authorization: The recordings came from a single private 1970s–80s oral-history collection controlled by its original interviewer, who authorized transcription and research use.No recording-time consent or deposit agreements addressed computational processing or third-party services.
  • Ethics review: No IRB review or independent human-subjects determination was obtained because the authors treated the work as secondary analysis of a private historical collection.They reported no intervention or interaction with individuals.
  • Third-party processing: Audio was processed through OpenAI and Google commercial transcription APIs under the institution’s organizational account and paid Google tier.The paper distinguishes endpoint-specific retention information from general API abuse-monitoring language.
  • Disclosure and data access: Names were redacted, but curated vocabulary and place-name examples retain residual reidentification risk in combination.The audio, database snapshot, and reference transcripts were not released.

11 Reproducibility and Data Availability

The preregistration and release materials are publicly available, with analysis scripts and preregistration documentation archived and mirrored.

  • The preregistration records frozen hypotheses, metric definitions, the statistical plan, and SHA-256 hashes of four analysis scripts.Three scripts are attached to the registration and mirrored in the component’s file storage.

12 Conclusion

In this preregistered paired ablation, full prompt-level context did not detectably improve side-level WER on the production-derived corpus. The gpt-4o result was concentrated near zero but could not resolve effects of that size reliably from one transcription per cell, while Gemini results were too unstable for a comparable negative inference.

  • Full prompt-level context produced no detectable improvement in side-level WER across 19 cassette sides from one production-derived oral-history collection.The conclusion comes from a preregistered paired ablation using the production transcription path.
  • Run-to-run pipeline variability exceeded the observed gpt-4o arm difference on four deliberately selected sides.Effects of that magnitude could not be resolved reliably from one realized transcription per cell.
  • The Gemini results were substantially less stable and did not support a comparable negative inference.The rerun indicated stochastic, arm-dependent pathological behavior on unstable sides.
  • The study’s null does not establish equivalence or provide a repeated-run population bound.The observed gpt-4o differences were concentrated near zero, but the study was not designed as an equivalence test.

Competing Interests

The authors report no financial interests in the providers and describe the study as independently conducted, with system operation, corpus creation, and reference transcription handled by the named authors.

  • Neither OpenAI nor Google funded, reviewed, or otherwise supported the work, and no author holds a financial interest in either provider.Provider access was purchased at list price under an institutional account.
  • The production system’s developer operated with AI for Altruism, while the system operator produced the corpus and reference transcripts.The developer received no compensation for this pro bono study.
Loading 2608.28875v1…