Source-linked AI summary
Mandarin Humorous Homophone Recognition and Disambiguation in Automatic Speech Recognition
Sicheng Jin, Jinghao Chen, Mostafa Shahin, Beena Ahmed, Aditya Joshi
TL;DR
Mandarin speech systems struggle to recognize and correct homophone-based wordplay, particularly when intended written forms are rare or context-dependent. This paper proposes a task-conditioned ASR adapter that detects potential spans, generates branch-specific candidates, and selects corrections semantically. Experiments show improvements for several homophone and contextual humourphone settings, while gains remain limited for full homophones.
Problem
Mandarin ASR systems and speech LLMs struggle to map similar pronunciations to semantically distinct written forms, especially for rare, context-dependent, or playful forms.
Method
The proposed task-conditioned adapter detects potential homophone spans, routes inputs through ordinary or humourphone branches, generates LLM candidates, and selects corrections with a semantic selector.
Results
The adapter improves Whisper-v3 on common and uncommon homophones, while contextual prompting reduces mean T-CER by 8.7% relative with Prompt 3.
Takeaways & Limitations
The framework supports speech-based humourphone recognition and intended-form recovery, but its effectiveness depends on ASR stability and prompt design.
Takeaways & Limitations
Prompting gains remain limited for full homophones because identical syllables and tones require contextual and pragmatic inference beyond phonetic deviation.
Abstract
from arXiv · showhide
Automatic mispronunciation detection and diagnosis (MDD) plays a crucial role in L2 Mandarin pronunciation learning. While end-to-end (E2E) based MDD methods have substantially improved phoneme-level detection accuracy, diagnostic feedback remains limited, as segmental and tonal errors are not explicitly separated. In this paper, we propose a phonological feature-based MDD framework that models both segmental and tonal attributes within a unified Wav2Vec2-CTC architecture. Experimental results show that the proposed method reduces the False Acceptance Rate (FAR) by 10.1% and the Diagnostic Error Rate (DER) by 23.6% compared with the phoneme-only baseline system. By decomposing phonemes into low-level phonological components, the proposed approach enables more detailed and interpretable diagnostic feedback for L2 learners.
1. Introduction
Mandarin homophone ambiguity makes speech-based humourphone recognition and correction difficult, especially for rare, context-dependent, or playful forms. The paper proposes a task-conditioned ASR adaptation framework that detects potential spans, generates candidates, and selects contextually appropriate corrections.
- Motivation: Mandarin ASR systems and speech LLMs struggle when similar pronunciations map to semantically distinct written forms.The difficulty increases for rare, context-dependent, or deliberately playful forms because models tend toward high-frequency lexical choices.
- Motivation: Over 62.2% of Chinese characters have reported homophonic features, supporting extensive rhetorical wordplay and humour.Mandarin’s tonal system and constrained syllable inventory contribute to this high homophone density.
- Motivation: Homophone-based wordplay includes identical and near homophones whose interpretation depends on phonetic, prosodic, syntactic, semantic, and pragmatic cues.HumourPhone denotes utterances that evoke a familiar word or scenario through phonetic ambiguity and introduce an unexpected semantic twist.
- Approach: The proposed framework identifies intentional speech-based humourphone wordplay and restores its intended written form.Ordinary homophone correction is evaluated as a secondary setting.
- Contributions: The framework contributes a humourphone-oriented ASR adapter, a target-aware evaluation protocol, and a humourphone dataset.The protocol measures recognition and correction within annotated regions.
2. Methodology
The proposed task-conditioned adapter routes Mandarin speech through ordinary-homophone or humourphone processing, generates constrained candidate transcriptions, and selects an output using semantic criteria. Its workflow combines span detection, branch-specific prompting, and semantic selection to support correction and humourphone recovery.
- Workflow: The system first obtains an initial ASR transcription, then selects an ordinary-homophone or humourphone branch using detector output.The ordinary branch targets candidate homophone spans, while the humourphone branch adds humour-oriented contextual cues.
- Evaluation Data: The evaluation set contains 80 synthesised Mandarin audio samples collected from Chinese internet wordplay and divided into Full Homophones and Near Homophones.Samples were manually inspected and filtered for unreliable pronunciation, ambiguity, inappropriate content, and nonhumorous cases.
- HumourPhone Prompting: The humourphone branch uses recurring patterns covering phonetic deviation, code-switching, punchlines, semantic clashes, visual wordplay, and anthropomorphic attribution.These prompt-level guidelines guide the emotion detector and humourphone prompt builder rather than defining strict linguistic categories.
- Architecture: The adapter contains a Homophone Detector, a branch-based LLM prompt builder, and a BERT-based semantic output selector.An Emotion Detector determines whether the utterance enters the homophone or humourphone branch.
- Homophone Detection: Potential spans are detected with PoS awareness, ranked using word frequency and contextual suspicion, and expanded with neighbouring PoS-tagged phrases.Masked sentences and tagged span information constrain candidate generation and reduce boundary-localisation errors.
- Candidate Selection: Generated candidates are scored by a BERT-based semantic selector, which balances semantic similarity against intentional incongruity and overcorrection risk.The homophone branch retains the original transcription within a 5% relative margin, while the humourphone branch favours lower-similarity candidates consistent with detected patterns.
3. Experiments and Results
Experiments evaluate homophone correction on merged Mandarin datasets and humourphone recovery under prompt variants. The adapter improves weaker ASR outputs and contextual prompting helps intentional wordplay, but gains remain limited for full homophones.
- 3.1. Data: The CN Dataset merges AS-3 and CV25-CN using pinyin-based homophone categories, while HumourPhone is evaluated separately because of its small scale.
- 3.3. Metrics: Evaluation reports target-span character error rate, Recovery Rate, Detection Rate, Mean Uncommon Recall, and semantic similarity measures.
- 3.4. Homophone Differentiation: 3.70%: prompt-only Qwen3-LLM decoding increases overall T-CER from 3.60%, indicating instruction-only decoding does not improve fine-grained correction.
- 3.4. Homophone Differentiation: 4.35 percentage points: Whisper-v3+Adapter reduces uncommon-form T-CER from 24.74% to 20.39%.On common forms, T-CER decreases from 6.43% to 5.07%.
- 3.4. Homophone Differentiation: 0.71 percentage points: Qwen3-ASR+Adapter reduces uncommon-form T-CER from 13.56% to 12.85%, but common-form T-CER rises from 2.19% to 3.42%.The results suggest stronger ASR systems need more conservative correction strategies.
- 3.5. HumourPhone Contextual Adaptation: 8.7% relative: prompt 3 reduces overall mean T-CER from 15.21% to 13.88% on HumourPhone.Near-homophone T-CER decreases by 16.5% relative, from 11.61% to 9.69%, while prompt 4 slightly worsens overall performance.
- 3.5. HumourPhone Contextual Adaptation: Prompting gains remain limited, especially for full homophones, where identical syllables and tones require contextual and pragmatic inference.The paper identifies additional HumourPhone training examples and confidence-aware correction as future directions.
4. Conclusion
The paper proposes a task-conditioned ASR adapter for homophone recognition and humourphone recovery. Experiments show standard ASR favors common homophone forms, while basic-prompt LLM decoding provides limited improvement.
- The task-conditioned ASR adapter targets both homophone recognition and humourphone recovery.