PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Korean Lipreading as Visual Speech Recognition: Transfer, Census and Adaptation on OLKAVS

Se Un Park, Hakjun Kim, Taehoon Roh, Junyoung Park

arXiv:2609.28988v1eess.AScs.CLcs.CV

TL;DR

Population-level VSR scores can mask individual users’ errors, while personal use may involve arbitrary camera angles and unscripted speech. The paper evaluates an English-initialized video-only Conformer on OLKAVS and examines speaker-level error, personalization, and camera views; it reaches less than half the published error rate, with substantial variation across speakers.

  • Problem

    Population-level benchmark scores may hide the spread of individual users’ errors, despite personal use involving arbitrary camera angles and unscripted sentences.

    Source: §1 · p. 1

  • Method

    The study evaluates an English-initialized hybrid CTC/attention Conformer on OLKAVS and measures speaker-level error, frontal-video personalization, and camera-view effects.

    Source: §1 · p. 1

  • Results

    The Conformer recognizes Korean at less than half the published error rate, with held-out per-speaker CERs between 1% and 52%.

    Source: §5 · p. 4

  • Takeaways & Limitations

    For an individual user, the per-speaker error distribution—not its population mean—is the relevant figure of merit.

    Source: §4 · p. 4

    The study covers one corpus and language, one architecture and training configuration, and voiced studio speech; mouthed speech may be a further shift.

    Source: §4 · p. 4

Abstract

from arXiv · show

We present a personalized Korean visual speech recognition (VSR) system and quantify, on the nine-camera OLKAVS corpus, the gap between the population-level benchmark score and an individual user's error. A video-only Conformer initialized from English-trained weights attains 9.95 - 12.19% character error rate (CER) under the corpus protocol against the published 26.64, and 19.00 - 21.52 on unseen wording. Per speaker, CER spans 1.0 to 52.2%, with seen wording lowering CER by 7.0 - 9.0 points and professional delivery and spontaneous speech raising it by 8.5 - 10.5 and 12.7 points. A low-rank adapter with 4.6% of the parameters, trained on 4 to 29 minutes of the user's frontal video, lowers the CER of twelve high-error speakers by 2.13 to 3.58 points, transfers to every camera without loss, and keeps 85% of the full fine-tuning gain at 12% of its cost to other speakers. Cameras above the mouth plane add about six CER points as a constant offset that training on all views keeps small.

1. INTRODUCTION

Population-level VSR scores can obscure individual users’ errors and do not capture the arbitrary camera angles and unscripted speech of personal use. The paper measures these gaps on OLKAVS, including benchmark performance, speaker-level factors, personalization, and camera elevation.

  • The English-initialized 232M-parameter hybrid CTC/attention Conformer reaches 9.95 - 12.19% CER against the published 26.64 on Korean sentence-level VSR.Source: §1 · p. 1
  • The study measures held-out speaker errors and separates the effects of wording, speaker type, and speech mode.Source: §1 · p. 1
  • Personalization is evaluated by video minutes, camera-view transfer, and cost to other speakers; frontal recordings transfer across cameras, and low-rank adaptation retains most fine-tuning gains at lower cost.Source: §1 · p. 1
  • Paired evaluations on identical utterances find that cameras above the mouth plane add about six CER points, while training on all views keeps the penalty at that level.Source: §1 · p. 1

2. CORPUS, MODEL AND PROTOCOL

The study uses speaker-disjoint OLKAVS data, an English-initialized video-only Conformer, and paired evaluation protocols for benchmark and speaker-level analyses. Its corpus and camera setup, model training, wording controls, and personalization budgets define the scope of those comparisons.

2.1. Corpus

  • OLKAVS contains 1,000 speakers across 12,000 studio recordings, with speaker-disjoint 893/107 train/validation partitions; the camera array lacks documented angles.Source: §2.1 · p. 1

2.2. Model and training

  • A video-only 232M-parameter model uses 96-pixel, 25-Hz frames, a Conformer, and shared CTC/attention jamo outputs, initialized from English visual-only Auto-AVSR; M9 trains on all views.Source: §2.2 · p. 2, Figure 1 · p. 2

2.3. Evaluation protocol

  • The protocol scores camera A with pooled CER and separates seen wording from sentences whose normalized text is absent from training; matched-duration sampling and paired bootstrap intervals support comparisons.Source: §2.3 · p. 2

2.4. Speaker-level evaluation and personalization protocol

  • Census scores base M9 for 106 speakers on three held-out recordings; personalization uses nested 4–29-minute frontal-video budgets and compares full or frozen-front-end fine-tuning with 4.6% LoRA-32.Source: §2.4 · p. 2

3. RESULTS

The results show strong Korean VSR benchmark performance alongside wide differences in speaker error, wording, delivery, and camera elevation. Low-rank personalization improves user CER and transfers across views, while all-view training keeps elevation penalties small.

3.1. Official protocol

  • 9.95% and 12.19% CER under the published protocol are 14.5–16.7 points below the published V-model; unseen-wording CER is 19.00% and 21.52%.The unseen-wording results remain below the published seen-subset figure.Source: §3.1 · p. 3

3.2. Per-speaker error distribution

  • Per-speaker CER ranges from 1.0% to 52.2%; professionals average 31.26% versus 6.17% for ordinary speakers, and M1 and M9 errors correlate at 0.989.The sorted census covers 106 held-out speakers; nine of M1’s ten hardest speakers are also among M9’s ten hardest.Source: §3.2 · p. 3, Figure 2 · p. 3
  • Seen wording lowers CER by 7.0 / 9.0 points for ordinary / professional speakers; professional delivery adds 8.5–10.5 points and spontaneous speech adds 12.7 points.The target case of spontaneous speech by ordinary speakers is absent from the corpus.Source: §3.2 · p. 3, Table 2 · p. 3

3.3. Personalization

  • 2.13 points of average adapter gain arrive at 7 minutes and 3.58 at 29 minutes; rank 32 retains 85% of FT-FE gain in a 44 MB per-user file.All twelve high-error speakers improve, and the gain saturates quickly.Source: §3.3 · pp. 3–4
  • Camera-A-trained LoRA-32 improves 60 of 68 other camera-speaker sets with gains of 4.09 / 3.37 / 3.45 points above / level with / below the mouth plane.The reported gain is the same across those camera bands.Source: §3.3 · p. 4
  • On 50 other speakers, FT-FE raises CER by 8.27 points on average, while rank-32 adapters raise it by 0.98 on average and never more than 2.50.The adapter’s reported average degradation is 12% of FT-FE’s; the per-user adapter is detached for other speakers in deployment.Source: §3.3 · p. 4
  • Encoder-only adapters match full-adapter gains (O/P = 2.09 / 2.57 versus 2.07 / 2.33 points), while decoder-only gains are 0.09 / 0.49.The comparison uses twelve speakers, 29 minutes, and joint decoding.Source: §3.3 · p. 4

3.4. Camera elevation

  • On unseen wording, elevated cameras add 5.77 CER points on average; matched-view elevation costs 6.51 points versus at most 2.03 for azimuth, while all-view training keeps the penalty small.Frontal-only M1 incurs 18.93 points on the upper band, compared with 5.77 for M9.Source: §3.4 · p. 4

4. DISCUSSION AND LIMITATIONS

The per-speaker distribution, rather than the population mean, is the relevant figure of merit for an individual user. Personalization benefits concentrate among speakers with the highest visual error, while elevated cameras retain an additive penalty and spontaneous speech remains difficult.

4. DISCUSSION AND LIMITATIONS

  • The per-speaker distribution should accompany the official score because it, not the mean, is the relevant figure of merit for an individual user.Source: §4 · p. 4
  • Personalization gains concentrate on speakers the base model recognizes worst for visual reasons, while professionals see relatively smaller gains because their errors stem from wording rather than face.Resampling by speaker rather than utterance confirms the finding with slightly different intervals.Source: §4 · p. 4
  • During training, camera-A CER falls from 50.9 to 26.1% while the upper-camera gap remains 5.2 - 7.0 points, supporting an additive rather than proportional penalty.A second seed reproduces the pattern.Source: §4 · p. 4
  • A speaker-specific decoder adapter and count-based language model do not solve the 12.7-point spontaneous-speech penalty; the paper identifies general spontaneous Korean language modeling as needed.Source: §4 · p. 4
  • The study covers one corpus, language, architecture, English-trained initialization, and training configuration, with run-to-run variation bounded by a second seed rather than a seed study.Source: §4 · p. 4
  • Elevations are qualitative because camera identity, distance, and elevation are confounded; speech mode is rule-inferred, personalization is frontal, and the data are voiced studio speech.Mouthed speech may constitute a further shift.Source: §4 · p. 4

5. CONCLUSION

An English-pretrained video-only Conformer recognizes Korean under the OLKAVS protocol at less than half the published error rate, while held-out speaker CER ranges widely. Personalization on a few minutes of frontal video improves high-error speakers across cameras, but spontaneous speech and real-world video remain targets for further work.

5. CONCLUSION

  • Held-out speakers have CERs from 1% to 52%, while the video-only Conformer recognizes Korean under the OLKAVS protocol at less than half the published error rate.Source: §5 · p. 4
  • For deployment, personalize frontally once, use a camera level with or below the mouth, and retain the encoder adapter as a per-user file.Spontaneous speech requires a language model of spontaneous speech, which a user's own video does not provide.Source: §5 · p. 4
  • Future work includes real-world videos, synthesized views and unseen wording, a neural language model of spontaneous Korean, and professional lipreader annotation and evaluation.Source: §5 · p. 4
  • The Korean WER on scripted studio speech cannot be directly compared with the 18.7% to 12.8% WER reported by the best published English visual-only systems on LRS3.Source: §5 · p. 4
Loading 2609.28988v1…