Source-linked AI summary
Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition
Hao Shi, Yun Liu, Xuehao Yang, Jun Liu, Chuanbo Hua, Xuanjun Chen, Lianbo Liu, Shiao Zhu, Zixiong Su
TL;DR
Conventional Japanese ASR can assign the same orthographic target to utterances with different realized readings, and post-hoc text-only G2P cannot reliably recover that distinction. Ruby-ASR uses span-bound orthographic–reading targets, with auxiliary mora-level CTC supervision. Across five Japanese benchmarks, it improves direct lexical-reading recognition while maintaining strong orthographic transcription.
Problem
Japanese orthographic transcripts can map different realized readings to the same target, while post-hoc text-only G2P cannot reliably recover which reading was spoken.
Source: §1 · p. 1Method
Ruby-ASR uses span-bound orthographic–reading targets under subtitle-style and verbatim-style conventions, with auxiliary mora-level CTC supervision.
Source: §1 · p. 1, §3.3 · p. 2Results
Across five Japanese benchmarks, Ruby-ASR improves direct lexical-reading recognition while maintaining strong orthographic and script-aware transcription; weighted Kana CER is 3.75% for Ruby-ASR-ver and 4.01% for Ruby-ASR-sub versus 5.74% for Qwen3-ASR.
Source: §5.2 · p. 3, §6 · p. 4Takeaways & Limitations
The checkpoints and inference code are publicly released.
Source: §1 · p. 1Ruby supervision does not establish that the autoregressive decoder relies causally on speech rather than preceding orthographic context.
Source: §3.2 · p. 2
Abstract
from arXiv · showhide
Conventional Japanese automatic speech recognition (ASR) is supervised by an orthographic transcript, although the same written form can correspond to different lexical readings realized in speech. Such utterances receive an identical target, so their reading distinction is absent from the supervision interface and cannot be recovered reliably by post-hoc text-only grapheme-to-phoneme conversion. We present Ruby-ASR, which refines the conventional target into a span-bound orthographic--lexical-reading sequence. Unlike separate full-sentence orthographic and phonological outputs, the ruby representation locally binds each written span to its realized reading and permits deterministic recovery of both views. We instantiate the target under subtitle-style and verbatim-style transcription conventions using a Qwen3-ASR backbone; a mora-level CTC objective provides auxiliary monotonic reading supervision. The experimental results across five Japanese benchmarks show that refining the recognition target can improve lexical-reading recovery without sacrificing readable orthographic transcription. We release the checkpoints and inference code.
1. INTRODUCTION
The introduction frames Japanese ASR’s orthographic targets as insufficient for preserving realized lexical readings, then presents span-bound Ruby-ASR as a structured target that retains both transcript views.
1. INTRODUCTION
- Different Japanese readings can share an orthographic transcript, so conventional ASR assigns them identical targets and text-only G2P cannot recover which reading was spoken reliably.Kana- or phoneme-only ASR preserves pronunciation but loses orthographic distinctions and readability.Source: §1 · p. 1
- Prior joint systems use separate utterance-level decoders or locally aligned grapheme–phoneme targets, while transcript-prompted annotation requires a ground-truth transcript.The remaining challenge is scaling speech-conditioned lexical-reading supervision while retaining a directly usable orthographic transcript.Source: §1 · p. 1
- Ruby-ASR binds written spans to their realized readings, enabling deterministic recovery of orthographic and lexical-reading transcripts under subtitle-style or verbatim-style conventions.A shared mora-level CTC branch supplies auxiliary monotonic supervision and can optionally produce a decoder-free reading hypothesis.Source: §1 · p. 1
- Across five Japanese benchmarks, Ruby-ASR improves direct lexical-reading recognition while maintaining competitive orthographic accuracy.The checkpoints and inference code are publicly released.Source: §1 · p. 1
2. BEYOND ORTHOGRAPHIC SUPERVISION
This section formalizes how orthographic supervision collapses distinct realized readings, compares supervision interfaces, and defines a diagnostic for text-only reading reconstruction.
2.1. Target-level lexical-reading collapse
- When two utterances share orthography but differ in realized reading, orthographic supervision maps them to one target; Ruby supervision distinguishes them by the orthography–reading pair.The section frames the distinction using utterance X, orthography O, and lexical reading R.Source: §2.1 · pp. 1–2
2.2. Supervision interfaces and prior joint targets
- Unlike separate utterance-level outputs, Ruby-ASR interleaves locally aligned written spans and readings, preserving deterministic projections of both transcript views.The comparison also contrasts post-ASR G2P, kana-only ASR, and transcript-prompted annotation, which requires ground-truth orthography at inference.Source: §2.2 · p. 2, Table 1 · p. 2
2.3. Text-only reconstruction gap
- The text-only reconstruction gap is measured by applying the selected G2P system to ground-truth orthography and comparing its reading with the reference.This diagnostic separates text-only reading reconstruction error from orthographic recognition error; the primary analysis uses independently prepared or manually verified reading references.Source: §2.3 · p. 2
3. RUBY-ASR
Ruby-ASR replaces orthographic-only supervision with a span-bound target that pairs written spans with their realized readings. It combines autoregressive Ruby generation with auxiliary mora-level CTC supervision and evidence-constrained target construction.
3.1. Speech–language backbone
- Ruby-ASR starts from Qwen3-ASR’s audio encoder and modality projector, then uses an autoregressive decoder; conventional supervision targets orthography alone.Under conventional supervision, the realized reading is not part of the output target.Source: §3.1 · p. 2
3.2. Span-bound Ruby target
- Ruby targets pair annotated orthographic spans with realized readings, while leaving unannotated material, including primarily kana, outside the pairs.Readings are normalized to hiragana, and okurigana remains outside the lexical-span annotation.Source: §3.2 · p. 2
- One Ruby sequence permits deterministic recovery of both orthographic and lexical-reading transcripts.The reading is an explicit speech-conditioned target, but its position after the orthographic span means the decoder may also use preceding orthographic context.Source: §3.2 · p. 2
3.3. Auxiliary mora-level regularization
- A mora-level CTC head supplies complementary monotonic reading supervision; Ruby generation remains the main inference path.The CTC branch can also produce an optional decoder-free kana hypothesis, but is treated primarily as auxiliary regularization.Source: §3.3 · p. 2
3.4. Automatic target construction
- Because source corpora generally lack speech-grounded readings matched to the large-scale ASR pool, target construction filters audio–text pairs and constrains reading labels to OpenJTalk candidate sets.Unresolved or structurally invalid samples are excluded from Ruby supervision.Source: §3.4 · pp. 2–3
4. EXPERIMENTAL SETUP
The experiments fine-tune two Ruby-ASR transcription variants on a large Japanese speech pool and evaluate them against established ASR systems across five benchmarks. The setup reports orthographic, script-aware, and lexical-reading metrics, with oracle G2P restricted to two benchmarks.
4.1. Training data and transcription conventions
- Training uses approximately 93.1 million audio–text segments (171.5 thousand hours), with separate subtitle-style and verbatim-style Ruby variants and a mora-level CTC branch.All parameters are fine-tuned for one epoch, and no evaluation audio from the five named benchmarks is used for training.Source: §4.1 · p. 3
4.2. Benchmarks and baselines
- Evaluation covers B5K, CSJ, Book, CV8, and TEDx, comparing Ruby-ASR with Whisper-Large-v3, Kotoba-v2, ReazonSpeech systems, Qwen3-ASR, and Kana-Whisper.The two Ruby variants use the same architecture and objective but follow different transcription conventions.Source: §4.2 · p. 3, §4.1 · p. 3
4.3. Metrics
- Raw CER measures script-preserving orthographic transcription, SA-CER compares kana regions by reading and kanji regions by orthography, and Kana CER measures lexical readings.Oracle G2P results are limited to B5K and CSJ, whose reading references were independently verified; the tables report CER metrics across five benchmarks and Kana CER for lexical-reading transcription.Source: §4.3 · p. 3, Table 2 · p. 3, Table 3 · p. 3
5. RESULTS
Across five benchmarks, Ruby-ASR improves direct lexical-reading recognition while maintaining strong orthographic and script-aware transcription. Diagnostics show where direct prediction helps and identify remaining limits in rare and unseen readings and subtitle-boundary evaluation.
5.1. Orthographic and script-aware recognition
- 8.51% weighted Raw CER for Ruby-ASR-sub and 6.59% weighted SA-CER for Ruby-ASR-ver are lowest, versus 10.78% and 8.59% for Qwen3-ASR.On JSUT-Book, Ruby-ASR-sub’s Raw CER is 23.21% versus 8.80% SA-CER, indicating much of the raw error reflects script realization rather than reading mismatch.Source: §5.1 · p. 3
5.2. Lexical-reading recognition
- 3.75% and 4.01% weighted Kana CER for Ruby-ASR-ver and Ruby-ASR-sub, respectively, compare with 5.74% for Qwen3-ASR, 5.64% for Rz-k2, and 4.66% for Kana-Whisper.Ruby-ASR-ver outperforms the standalone mora-CTC path, reducing weighted error from 7.31% to 3.75%; Kana-Whisper remains best on B5K and CSJ, and Whisper+G2P on JSUT-Book.Source: §5.2 · p. 3
5.3. Oracle-orthography G2P diagnostic
- Oracle G2P retains 1.69% CER on B5K and 3.47% on CSJ, while Ruby-ASR beats it on B5K but trails on CSJ.Ruby-ASR-ver and Ruby-ASR-sub beat oracle G2P by 0.61 and 0.37 points on B5K, but trail by 1.94 and 1.90 points on CSJ; this unmatched comparison is diagnostic, not causal.Source: §5.3 · p. 4, Table 4 · p. 4
5.4. G2P-hard reading analysis
- On G2P-hard utterances, Ruby-ASR-ver lowers B5K Kana CER to 1.59% from 6.07% for oracle G2P and 4.79% for Rz-k2+G2P.B5K has 1,124/5,000 (22.5%) and CSJ 2,579/8,460 (30.5%) such utterances; on CSJ, oracle G2P has lower CER than Ruby-ASR, but Ruby models more often recover the complete utterance reading.Source: §5.4 · p. 4, Table 5 · p. 4
5.5. Conditional and ambiguity-sensitive reading accuracy
- Both Ruby variants exactly recover about 97% of readings when the orthographic span is correct, including on multi-reading spans.Degradation on rare and unseen pairs points to lexical coverage as the main remaining challenge, especially on spontaneous CSJ; those subsets contain only 245 and 116 spans.Source: §5.5 · p. 4, Table 6 · p. 4
5.6. Subtitle-boundary diagnostic
- Ruby-ASR-sub has the second-lowest total subtitle-boundary overflow at 0.57%, but the references are too unreliable for content-level scoring.Boundary overflow measures segment-level output control rather than hallucination or recognition accuracy.Source: §5.6 · p. 4, Table 7 · p. 4
6. CONCLUSION
Ruby-ASR refines Japanese ASR supervision with span-bound ruby targets that preserve orthographic identity and lexical reading together. Across five benchmarks it improves direct reading recognition while maintaining strong orthographic transcription, with rare and unseen pairs remaining a major limitation.
6. CONCLUSION
- Ruby-ASR replaces conventional orthographic targets with span-bound ruby targets that retain each written span and its realized reading in one structured sequence.The sequence permits deterministic recovery of conventional orthographic and lexical-reading transcripts.Source: §6 · p. 4
- Across five benchmarks, direct lexical-reading recognition improves while orthographic and script-aware transcription remains strong.Source: §6 · p. 4
- Direct prediction is particularly effective on B5K cases where the selected G2P fails, while rare and unseen orthography–reading pairs remain a major limitation.Source: §6 · p. 4
- Released models provide structured Ruby generation and optional decoder-free mora output for Japanese speech and reading-aware research.Source: §6 · p. 4