HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition

1 Introduction

• The benchmark addresses the underexplored question of whether strong monolingual ASR performance extends to utterances mixing languages.The gap is especially salient for Korean-English, a typologically distant language pair.Source: §1 · p. 1
• The paper compares natural word- and phrase-level code-switching data with synthetic sentence-level data for fine-tuning ASR models.The benchmark covers 1,121 high-quality utterances across various topics.Source: §1 · p. 1
• HiKE is a public Korean-English code-switching benchmark with loanword labels and word-, phrase-, and sentence-level annotations.The paper uses these labels to assess how 10 multilingual ASR models handle different types of code-switching.Source: §1 · pp. 1–2

2.1 Automatic Speech Recognition

• ASR research has moved from streaming CTC systems to nonstreaming Transformer encoder-decoders and more recently to pretrained LLMs.The cited discussion associates Transformers with full audio context and LLMs with high accuracy and task extensibility, alongside substantial computational needs.Source: §2.1 · p. 2

2.2 Code-Switching Speech Datasets

• Public code-switching datasets are rare, and scarcity is particularly acute for typologically distant Korean-English.An existing government-funded Korean dataset is restricted to Korean nationals and inaccessible to the global research community.Source: §2.2 · p. 2
• Prior work has generated synthetic CS data by concatenating monolingual utterances or using text-to-speech models [Yan et al., 2025].Concatenation can have mid-utterance vocal discontinuities and is limited to sentence-level CS; TTS data may have poor audio quality and miss real-world CS variability.Source: §2.2 · p. 2
• HiKE is presented as the first globally accessible, non-synthetic Korean-English CS benchmark.The paper positions it against synthetic data that may not capture authentic CS patterns and acoustic variability.Source: §2.2 · p. 2

3.1 Script Writing & Cloning

• The dataset was built from 575 manually authored seed scripts across eight topics, each cloned once with Claude 3.5 Sonnet and then manually reviewed and corrected.The process was designed to support high quality while minimizing human effort.Source: §3.1 · p. 2

3.2 Recording

• Thirteen bilingual Korean-English speakers recorded scripts, yielding 1,150 recordings; after 29 were discarded, the curated dataset contained 1,121 utterances totaling about 2.2 hours.Speakers recorded 50 to 100 scripts each in a quiet environment using personal devices and a web-based interface.Source: §3.2 · pp. 2–3

3.3 Metrics

• MER evaluates the whole utterance, while PIER evaluates words at code-switching transition points.For PIER, the authors tagged words where switching occurs to assess whether models switch languages accurately at those points.Source: §3.3 · p. 3

3.4 Hierarchical CS-Level

• The hierarchical scheme distinguishes sentence-, word-, and phrase-level CS to analyze how models handle intra-sentential switching between typologically distant languages.Phrase-level CS can introduce irregular structures such as altered word order, while sentence-level switching is more predictable.Source: §3.4 · p. 3

3.5 Loanwords Post-processing

• Loanword labels let evaluation accept either Korean or English forms of dataset loanwords, reducing measurement noise by an average of 4.9% in MER and 7.9% in PIER.Without this treatment, a valid alternate-script transcription could be penalized; the labels address that ambiguity.Source: §3.5 · p. 3

4.1 CS-ASR of Multilingual ASR Models

• MER increased by a factor of 3 to 13 on CS data versus monolingual performance across all 10 evaluated multilingual ASR models.The models included CTC-, Transformer-, and LLM-based systems.Source: §4.1 · pp. 3–4
• SENSEVOICE-SMALL had better overall MER than similarly sized WHISPER-SMALL, yet showed high error rates at code-switch points.The paper notes that CTC models may be robust in overall transcription but struggle at language transitions.Source: §4.1 · p. 4
• GPT-4O-TRANSCRIBE was the only LLM-based model to outperform WHISPER-LARGE; GEMMA-3N and AUDIO-FLAMINGO-3 had both MER and PIER above 54.Unlike most non-LLM models, GPT-4O-TRANSCRIBE performed best on word-level CS and worst on sentence-level CS; the authors hypothesize this reflects its training-data distribution.Source: §4.1 · p. 4
• Whisper-family CS-ASR capability emerged with increasing model size, but even the largest model’s CS error rate remained over five times its monolingual rate.The authors conclude that scaling alone is insufficient for practical CS-ASR performance.Source: §4.1 · p. 4

4.2.1 Experimental Details

• Fine-tuning WHISPER-MEDIUM used natural intra-sentential CS data, synthetic sentence-level CS data made by concatenating FLEURS and Common Voice utterances, or both.The natural data came from AIHub and covered word- and phrase-level CS.Source: §4.2.1 · p. 4

4.2.2 Results

• More than 6.8% improvement in both overall MER and PIER followed fine-tuning on synthetic CS data alone; natural intra-sentential data improved almost all metrics more, except sentence-level MER and PIER.The synthetic data were created by concatenating monolingual utterances; natural data performed better on the other reported metrics.Source: §4.2.2 · p. 4, Table 3 · p. 4

4.3 Qualitative Results

• The observed error types were phonetic transcription into the other language’s script, instruction-following failures, and hallucinated repetitive or excessive content absent from the audio.Instruction-following failures occurred in multitask models and were especially pronounced in AUDIO-FLAMINGO-3.Source: §4.3 · p. 5, Figure 2 · p. 5
• Instruction-following failures and hallucinations increased markedly on CS data, while phonetic transcription errors also occurred with monolingual data; the authors attribute the CS increase to scarce CS exposure during training.The authors link that scarcity to difficulty generalizing ASR performance to CS scenarios.Source: §4.3 · p. 5

5 Conclusion

• HiKE is presented as the first public high-quality hierarchical Korean-English CS-ASR benchmark with hierarchical CS-level and loanword labels.The benchmark’s labels cover word-, phrase-, and sentence-level CS.Source: §5 · p. 5
• Across 10 multilingual models, strong monolingual performance did not generalize to CS, and word- and phrase-level CS were less accurately recognized than more predictable sentence-level CS.The conclusion associates the harder levels with dense, irregular switch points and complex grammatical structures.Source: §5 · p. 5
• Fine-tuning improved CS-ASR capabilities, including with synthetic inter-sentential data made by concatenating monolingual utterances.The conclusion identifies extending research to other language pairs, synthetic CS generation, and generalization analysis as future directions.Source: §5 · p. 5

• The scarcity of Korean-English ASR models precluded broad comparisons across diverse architectures.Source: Limitations · p. 5
• Only GPT-4O-TRANSCRIBE yielded meaningful code-switching performance among the LLM-based models, preventing reliable analysis of their common characteristics.Source: Limitations · p. 5
• Scarce high-quality code-switching data restricted fine-tuning experiments to a small scale, precluding thorough investigation of approaches such as scaling up synthetic data.Source: Limitations · p. 5

A Experimental Setup

• Unless otherwise specified, experiment results came from a single run on an NVIDIA RTX 6000 Ada GPU.Source: A Experimental Setup · p. 7
• Experiments used PyTorch 2.8.0 and Transformers 4.56.2.Source: A Experimental Setup · p. 7
• Fine-tuning ran for approximately one epoch on a single A100 GPU with batch size 16, cosine annealing, 10% warmup, and peak learning rate 1e −5.Source: A Experimental Setup · p. 7

B Dataset Recording

• The web interface provided recording instructions and scripts in both Korean and English.Source: B Dataset Recording · p. 7
• Participants could review their recordings and make retakes through the web interface.Source: B Dataset Recording · p. 7
• Table 4 summarizes participant gender and educational-background distributions.Source: B Dataset Recording · p. 7

C License

• The experiments used the official PIER implementation under the Apache 2.0 License.Source: C License · p. 8
• The authors planned to release their HiKE evaluation code under the Apache 2.0 License.Source: C License · p. 8

D Script Cloning Prompt

• Figure 4 shows the prompt used to clone a new script from an original script.Source: D Script Cloning Prompt · p. 8

E CS-Level Determination

• Sentence-level code-switching applies when a sample contains at least one entirely English sentence and one entirely Korean sentence.Source: E CS-Level Determination · p. 8
• Phrase-level code-switching applies to remaining samples with two or more consecutive words in a language different from the rest.Source: E CS-Level Determination · p. 8
• Word-level code-switching applies to remaining samples where the switch consists of only one non-consecutive word; Figure 5 presents the procedure.Source: E CS-Level Determination · p. 8, Figure 5 · p. 9