PaperlayerView exact revision on arXiv

Source-linked AI summary

Building a Production Greek-English Speech Recognizer

Christos Petrocheilos, Cleopatra Papadopoulou, Chris Porikis, Ioakeim Perros, Ayoub Kirouane, Themistoklis Nikolis

arXiv:2609.13498v1eess.AScs.AIcs.SD

TL;DR

Sophea addresses production Greek-English speech recognition in noisy settings and sentences that switch languages. The authors evaluate it against nine production gates, find that no tested training-data composition satisfies them all, and respond with a family of models and serving interventions.

  • Problem

    Production recognition for Greek-English speech that switches languages and is recorded over telephone lines or in crowded rooms remains an engineering challenge.

    Source: §1 · pp. 1–2

  • Method

    The authors build and evaluate Sophea against nine production gates and use controlled exposure experiments to study the bilingual training trade-off.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    No training-data composition passed all nine gates; the Greek noisy-environment target required about 1,500 steps, versus English language-identification tolerance of about 250 baseline or 1,250 rebalanced steps.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The team shipped a family of models, a serving layer that routes between them, and decode-time interventions rather than one model.

    Source: §1 · p. 2

    The paper releases no model weights, training data, evaluation sets, or internal tooling.

    Source: §10 · p. 17

Abstract

from arXiv · show

We report a multi-month engineering program to build Sophea, a production bilingual Greek-English automatic speech recognition system. We evaluate the system against nine production gates covering Greek and English word error rate, language identification, and hallucinations on non-speech audio. Across twenty-three training iterations and two model architectures, no training-data composition passed all nine gates simultaneously. Meeting the Greek noisy-environment target required about 1,500 steps of dense domain exposure, while preserving English language identification tolerated only about 250 steps, or about 1,250 with a rebalanced mix that reduced Greek accuracy. We describe a six-stage data pipeline in which calibrating an audio-quality filter against in-domain anchors reduced the discarded share of scored Greek audio from 98.7 percent to 10.6 percent. A pre-registered ablation isolated a hallucination defect to one training-data package. A three-model ROVER ensemble increased gate coverage from 4-7 of 9 for individual models to 9 of 9 and reduced overlapping-speech WER from 53.35 percent to 37.87 percent, a 29 percent relative improvement. A separate learned per-clip arbiter over two models is listed as sophea/asr-k1 (preview) on the public Open ASR Leaderboard, with 4.26 percent average WER across eight public English test sets, and reaches 25.88 percent WER on live Greek noisy-environment traffic. We also document five cases in which a measurement tool produced a plausible but incorrect result and seven substantial approaches that were evaluated but not shipped. No model weights or training data are released; we report methodology and quantitative results only.

1 Introduction

Sophea targets bilingual Greek-English recognition in noisy, mixed-language speech, where the paper finds that no training-data composition tested for the small model passed all nine production gates. The introduction situates its engineering program in established ASR architectures, evaluation resources, and multilingual capacity trade-offs.

1 Introduction

  • Sophea targets Greek and English recognition across noisy environments, business meetings, and mid-utterance language switching, evaluated against nine customer-facing production gates.Three products shaped the program: meeting summarization, robot voice handling, and a latency-sensitive voice-to-text feature.Source: §1 · p. 2
  • No training-data composition passed all nine gates for the small model; Greek and English requirements demanded incompatible exposure budgets.The paper says composition changes move a run along the measured frontier, while a larger model, auxiliary objective, different architecture, or untested schedule might move it.Source: §1 · p. 2

2.1 ASR Architectures and Ensembling

  • The production system draws on Whisper and Qwen lineages, while its three-model ensemble combines the production recognizer, Greek-adapted Canary, and LoRA-adapted Whisper using ROVER word voting.The paper also evaluates Sortformer for detecting overlapping speakers.Source: §2.1 · p. 3

2.2 Evaluation Resources and Calibration

  • Evaluation spans public multilingual and English benchmarks alongside proprietary Greek call-center, meeting, and phone-dictation benchmarks described qualitatively.Meeting-domain and overlap evaluation also uses the AMI Meeting Corpus.Source: §2.2 · p. 3

2.3 Multilingual Training Trade-offs

  • The paper reports a discrete training-step-budget frontier between two ASR languages, while noting that multilingual capacity competition itself is known from language modeling and translation.Prior work documents a “curse of multilinguality” and capacity competition in neural machine translation.Source: §2.3 · p. 3

3 The Bilingual Trade-off

Controlled exposure experiments show that the tested small model cannot satisfy Greek noisy-environment and English language-identification gates with one training-data mix. Acoustic-neighborhood tests and the resulting serving design clarify how the team responded.

3 The Bilingual Trade-off

  • The exposure frontier comes from single runs without repeated seeds, leaving run-to-run uncertainty unquantified and limiting the impossibility result to the tested scale and configuration.The experiments varied exposure quantity and mix composition; whether capacity, objective, or schedule changes move the frontier remains undistinguished.Source: §3 · p. 4
  • 835 hours of clean studio-quality English protected no noisy-environment accuracy, while 577 hours of noisy, overlapping meeting-style English held it over the same training steps.The authors conclude that data protects a gate when it shares the acoustic neighborhood of the interference.Source: §3 · p. 4
  • The team shipped a serving layer that routes calls and meetings to specialized checkpoints, uses forced-language decoding in known Greek-heavy domains, and otherwise relies on implicit language identification.The routing decision followed the first seven model versions.Source: §3 · p. 4
  • The Greek noisy-environment gate requires ≥1,500 exposure steps, while English language identification tolerates ≤250 baseline or ≤1,250 rebalanced steps, so the intervals do not overlap.The rebalanced mix stalls Greek accuracy short of its gate; the measured frontier is for the tested scale and configuration.Source: §3 · p. 4, Figure 1 · p. 5

4 Mining the Data

The data section builds measurement and filtering safeguards around six pipeline stages, with in-domain calibration preventing an absolute audio-quality threshold from discarding most scored Greek audio. It also reports normalization, contamination checks, and an ablation of a hallucination-inducing data package.

4.1 A Six-Stage Quality Pipeline

  • The six-stage data pipeline filters audio quality, words-per-minute plausibility, transcription confidence, teacher agreement, forced alignment, and final language parity.One representative finished pool contained 1,073,294 Greek rows and 1,079,148 English rows, a 0.55% parity delta.Source: §4.1 · p. 5
  • Words-per-minute and forced-alignment checks target transcript-audio mismatches that previously taught fluent hallucinations; subsequent evaluation batteries recorded zero hallucinations.An earlier pool had roughly 40,000 mismatched rows and pushed one internal hallucination benchmark over 100.Source: §4.1 · p. 5

4.2 Calibrating the Quality Filter

  • 10.6% of scored Greek rows were dropped with the calibrated audio-quality threshold, versus 98.7% with the conventional absolute threshold.The calibrated threshold used known-good in-domain references; the conventional threshold would also have discarded the clean FLEURS benchmark.Source: §4.2 · pp. 5–6, Figure 2 · p. 6

4.3 A Greek-Aware Normalizer

  • The Greek-aware normalizer folds diacritics and case, removes fillers in both languages, and converts spoken Greek numbers, units, and percentages consistently in references and hypotheses.Unit and percentage phrases are canonicalized before the numeric grammar runs.Source: §4.3 · p. 6

4.4 Contamination Checking and a Pre-Registered Ablation

  • Raw-waveform cross-correlation with a half-second lag tolerance found zero true duplicates among 227 held-out meeting clips after two acoustic-similarity checks produced many false matches.The false matches arose because the earlier checks measured general meeting similarity rather than recording duplication.Source: §4.4 · pp. 6–7
  • Removing the short-utterance and empty-target-negative package brought hallucinated phrases to 20, within the healthy baseline of 17 to 18; removing noise alone left 24.The pre-registered ablation varied candidate ingredients while holding hyperparameters fixed; the base checkpoint lineage was cleared.Source: §4.4 · p. 7

5 The Model Development Campaign

The campaign pursued separate compact and larger Whisper model lines, followed by a stock-weight turbo tier. The iterations produced partial gate wins and decode-time fixes.

5.1 The Compact Model, Seven Iterations

  • C6 first reached seven of nine gates after a label-only change; C7's best noisy-environment WER improved to 26.32% but missed the ≤25.25 gate.Across C1–C5, the best result was six of nine gates; C7 did not raise the overall pass count beyond C6.Source: §5.1 · p. 7, Figure 3 · p. 8

5.2 The Large Model, Seven Iterations

  • L1–L3 traced the boilerplate bug to an interaction between pseudo-labeled Greek data normalization and upsampling, while adding meeting data still left a separate read-Greek regression.The initial single-variable attribution had focused on the wrong source of the bug.Source: §5.2 · p. 8
  • L4 shipped as the first production model in the larger-model line, passing five of six hard gates with English unchanged.These were campaign-specific internal ship-decision gates, distinct from the nine production gates.Source: §5.2 · p. 8
  • L5 was aborted after a pre-registered stability check fired, but an audit found its baseline assumption was wrong; no checkpoint shipped.The check treated a low-level defect already present in the untrained base model as absent.Source: §5.2 · p. 8
  • L7 reduced boilerplate hits from 26 to 0 by removing “ok” and “okay” from the filter, with other metrics unchanged to two decimal places.It shipped as an additive decode-time configuration on L4, with no new weights.Source: §5.2 · pp. 8–9

5.3 The Turbo Tier

  • Decode-time language identification, a wider beam, and a repetition penalty lowered turbo's code-switched WER from 10.78 to 9.79 without training.Turbo uses stock weights; it is competitive on clean read speech but falls behind on informal or community-sourced Greek.Source: §5.3 · p. 9

6 Ensembling and Overlapping Speech

The paper evaluates word-level voting and learned per-clip routing as distinct ways to combine recognizers, reporting production-gate, public-English, live Greek-English, and overlapping-speech results. It also explains why overlap voting was not promoted, citing deployment and evaluation boundaries.

6 Ensembling and Overlapping Speech

  • ROVER aligns three recognizers’ hypotheses into a word transition network and selects the highest-vote candidate at each aligned position, breaking ties by average per-word confidence.The candidates include a null/deletion symbol.Source: §6 · p. 10
  • Voting between the two strongest models passed 8 of 9 gates; suppressing output when the strongest detected silence closed the gap to 9 of 9 in a live system with automatic language routing and no oracle hint.The silence-suppression rule was found after the fact and recorded as conditional, not as part of the headline result.Source: §6 · p. 10

6.1 A Second Combination Strategy: Sophea ASR K1

  • K1 routes each clip’s entire output to one of two models using a learned classifier, rather than blending words; leave-one-set-out checks beat both models on every held-out set.The arbiter was fit on development-pool validation splits, not leaderboard test data; K1 serves offline batch transcription.Source: §6.1 · p. 11
  • 4.26 average WER across eight public English test sets is the live-board listing for Sophea ASR K1 (preview).K1’s seven cleaned-set average is 4.35; the eight-set board figure also includes Voice Arena Monsoon.Source: §6.1 · p. 12, Table 4 · p. 13
  • 25.88% WER on Greek noisy-environment audio was K1’s single-served-model pass of the ≤26 gate, while English noisy-environment WER improved from 15.14% to 14.21%.The Greek leg uses the far-field Qwen variant alone; on the same Greek set, the large model scored 24.63%.Source: §6.1 · p. 12

6.2 Overlapping Speech

  • 53.35% to 37.87% WER on overlap-only audio was a 29% relative reduction across eighteen real meetings, without breaking another gate.The production model’s overlap failure is primarily deletion: it misses most words from the speaker it locks onto second.Source: §6.2 · p. 14, Figure 6 · p. 14

6.3 Why We Did Not Promote This

  • Overlap voting was not promoted because overlap is rare in the meeting data, available benchmarks are English-only, and production streaming latency was unmeasured.The evaluated overlap detector’s localization precision peaked at 0.30 to 0.39 against a required 0.85; the ensemble member also had confirmed AMI pretraining exposure.Source: §6.3 · p. 14

7 Five Ways Our Instruments Lied

Five measurement failures produced plausible but incorrect results; each was caught only after a second instrument disagreed. The cases span training, benchmarking, contamination checks, and evaluation infrastructure.

1. Curriculum ordering

  • Three restarts misdiagnosed a six-fold loss jump; testing stationary shuffled data showed curriculum ordering, not a discrete bug, caused it.The easy-to-hard ordering combined with a decaying learning rate preceded the jump.Source: §7 · p. 15

2. Loss logging

  • 32 times larger than it looked, the logged SFT loss omitted gradient-accumulation scaling, while gradients were also unscaled and clipping bound on every run.The apparently stalled curve was actually a healthy slow continuation.Source: §7 · p. 15

3. Benchmark regression

  • 4.46 to 8.42 WER on one everyday-speech benchmark came from a spoken phone number rendered as digits, not a broad quality regression.The other nine clips were marginally better; the authors retired the gate as inverse-text-normalization-sensitive.Source: §7 · p. 15

4. Contamination checks

  • Thousands of apparent contamination matches were false positives from acoustic-similarity methods; waveform cross-correlation found zero duplicates.The methods detected general similarity between meetings rather than literal recording duplication.Source: §7 · p. 15

5. Evaluation harness

  • A parallel evaluation race silently overwrote one ablation arm; a clean re-evaluation reduced the apparent effect by an order of magnitude, within single-clip noise.The corrected result voided the original claim without changing the ablation’s main conclusion.Source: §7 · p. 15

8 Negative Results

The section records seven substantial approaches that were evaluated but not shipped, alongside pre-registered plans, measured outcomes, and closed verdicts. Results range from unavailable data to regressions and gains too narrow to generalize.

Noisy-environment under-generation

  • Noisy-environment Greek under-generation remained unsolved: deletion-targeted tuning worsened length ratio, reward tuning passed parity without closing the error gap, and fusion regressed another benchmark.The three-encoder fusion moved the target metric by −0.37 and the independent meeting benchmark by +3.9; the Section 6 ensemble remained the production answer.Source: §8 · pp. 15–16

New English training data

  • No fresh English meeting or far-field corpus was usable; the best substitute held every gate and improved nothing above threshold.Candidates were already in the training pool, blocked by licensing, or unavailable in usable form.Source: §8 · p. 16

Text-only error correction

  • Text-only correction regressed net quality at every checkpoint, correcting some errors while inventing others.The character-level model was trained on hypothesis-reference pairs.Source: §8 · p. 16

Audio-conditioned error correction

  • Audio-and-text-conditioned deliberation also regressed quality at every checkpoint despite an explicit anti-overcorrection training guard.Source: §8 · p. 16

Spectrogram-as-image transcription

  • A blank grey image scored better than the true spectrogram, while permuting frequency bands improved WER, disproving spectrogram-as-image transcription under the adversarial controls.The authors conclude the model was reciting a learned Greek language prior rather than reading the image.Source: §8 · p. 16

Direct meeting-audio fine-tuning

  • Direct meeting-audio fine-tuning improved held-out accuracy on that meeting set but breached the clean-Greek-read-speech gate and worsened an independent meeting benchmark.The authors conclude it learned one meeting’s conventions rather than meetings generally.Source: §8 · p. 16

Turbo-tier fine-tuning

  • Two turbo-tier fine-tuning attempts regressed code-switching further below the untrained baseline, so turbo ships with stock weights and decode-time tuning.The shipped configuration includes the code-switching fix described in Section 5.3.Source: §8 · p. 16

9 What to Take Away

The paper argues that Greek-English production ASR requires recognizing a trade-off frontier rather than endlessly changing data mixtures. It recommends calibrated measurement, targeted decode-time fixes, pre-registered ablations, and transparent reporting of approaches that did not ship.

9 What to Take Away

  • A trade-off frontier is not a data problem: composition moves performance along it, while capacity or objective changes may move it.The authors leave open whether schedule or ordering changes could move the frontier.Source: §9 · p. 16
  • Calibrate automatic scorers against known-good in-domain references before trusting thresholds, which may otherwise discard most real data.The warning concerns textbook cutoffs built for clean, well-resourced languages.Source: §9 · p. 16
  • Three program cases were ultimately solved with decode-time configuration changes after retraining was tried first, but this is not a validated general rule.The cases involved silence hallucination, code-switching weakness, and boilerplate output.Source: §9 · p. 17
  • Pre-registered ablations and audits of the evaluation harness helped preserve confidence in a conclusion despite a measurement-code bug.The design specified in advance what would confirm or exonerate each candidate cause.Source: §9 · p. 17
  • Seven substantial approaches did not ship; their measured outcomes and blocking constraints were documented for possible reconsideration if those constraints change.Constraints included a missing dataset, an overcorrecting architecture, and an insufficiently precise detector.Source: §9 · p. 17

10 Availability

The paper reports methods and measured outcomes but releases no model weights, training data, evaluation sets, or internal tooling. The K1 system can be tested through a public API, and Table 4 is the fully public, reproducible comparison.

10 Availability

  • No model weights, training data, evaluation sets, or internal tooling are released; K1 is available for testing through a public API.The paper reports methodology and measured outcomes for reuse by teams working on other under-served language pairs.Source: §10 · p. 17
  • Proprietary Greek meeting and noisy-environment benchmarks are not independently reproducible, while Table 4 is fully reproducible against the public leaderboard resources.The public-benchmark portions of Table 2 and Figure 4, AMI-derived Figure 6 numbers, and Figure 5 are also reproducible against public resources.Source: §10 · p. 17

11 What We Are Not Claiming

The authors limit the trade-off finding to one model scale and training configuration, and describe boundaries on reproducibility and evaluation scope. They also disclose AI assistance in drafting and state that authors checked the reported numbers and take responsibility for the paper.

11 What We Are Not Claiming

  • The reported impossibility result is measured at one model scale and training configuration, not established for larger scales or an auxiliary language-identification objective.Several meeting and noisy-environment evaluation surfaces are proprietary and not independently reproducible.Source: §11 · p. 17
  • The overlapping-speech result is measured on English meeting audio, has a known contamination caveat on one ensemble member, and does not extend to Greek.The ensemble and K1 use an earlier Whisper-based checkpoint rather than current production checkpoint L4; the effect of substituting L4 has not been tested.Source: §11 · pp. 17–18
  • Tables 3 and 4 are single-run point estimates without confidence intervals, so differences below roughly one point are not treated as established.Weights, code, and data are not released, limiting independent replication of the exact reported numbers.Source: §11 · p. 18

12 AI Disclosure

  • AI assistance produced the initial manuscript draft from internal records, but authors reviewed technical accuracy, verified numeric claims, and take responsibility for the content.The disclosure says the AI system independently generated no experimental data, analysis, or conclusions.Source: §12 · p. 18

Appendix A: Gate and Metric Definitions

The appendices define the production gates and metrics, then specify the evaluation surfaces and measurement procedures behind the reported results. They also distinguish public leaderboard evaluation from internal or domain-specific measurements.

Appendix A: Gate and Metric Definitions

  • Confidence intervals use clip-level bootstrap resampling unless stated otherwise, while real-time factor measures API throughput rather than per-utterance latency.The stated interval uses 1,000 resamples; real-time factor above 1.0 means audio is processed faster than it plays.Source: Appendix A · p. 20
  • The nine gates comprise four Greek and three English WER ceilings, a 95% language-identification floor, and zero non-empty transcripts on silence and noise clips.WER counts substitutions, deletions, and insertions after minimum-edit alignment; both hypothesis and reference use the Greek-aware normalizer, while language identification is measured without a language hint.Source: Appendix A · pp. 19–20

Appendix B: What Each Number Was Measured On

  • The bilingual trade-off used controlled runs from a common checkpoint varying Greek noisy-environment exposure; the ablation used three matched 750-step arms varying only mix composition.The trade-off figure plots required and tolerated step-budget intervals, while boilerplate was measured on a fixed probe at checkpoint 750.Source: Appendix B · pp. 20–21
  • UTMOS drop rates came from one scored Greek training-pool build, and gate-progress counts used each version’s best checkpoint on the fixed evaluation battery.L4 ship-decision values came from single-checkpoint scores on separate frozen internal probes, not the nine production-gate evaluation surfaces.Source: Appendix B · p. 20
  • Large-versus-turbo comparisons use standard public FLEURS and LibriSpeech test splits plus a fixed internal Greek evaluation set; the three-recognizer table reports single-run scores on frozen per-row batteries.The recognizer columns share one normalizer, and the table has no confidence intervals.Source: Appendix B · p. 20
  • K1 leaderboard scores and Figure 5 bars use the leaderboard’s own single-run harness, while the live noisy-environment figures come from the production API.An independent re-decode agreed on 712 of 719 clips (99.0%) and scored 14.15 rather than 14.21, a disclosed spread of about 0.06 WER points.Source: Appendix B · pp. 20–21
  • The overlap-fusion result uses 18 real AMI meeting recordings and computes WER on overlap-only speech spans for a single model versus a two-model confusion-network vote.The reported interval comes from clip-level bootstrap resampling.Source: Appendix B · p. 21

Appendix C: Decode-Time Configuration

Decode-time configuration suppresses non-speech output, filters audited boilerplate phrases, and routes recordings according to language and domain.

Appendix C: Decode-Time Configuration

  • A voice-activity detector and two calibrated confidence thresholds jointly suppress decoding on frames flagged as non-speech.The thresholds use no-speech probability and log probability.Source: Appendix C · p. 21
  • The marker-phrase filter uses an audited list that excludes phrases occurring as legitimate content in real recordings.Section 5.2 found two such phrases contributed zero true positives and all filter false positives.Source: Appendix C · p. 21
  • Calls and meetings are routed by expected language and domain, with forced-language decoding for known Greek-heavy domains and implicit language identification otherwise.Long recordings use overlapping chunks, and known entities receive decoding bias where available.Source: Appendix C · p. 21
Loading 2609.13498v1…