PaperlayerView exact revision on arXiv

Source-linked AI summary

ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context

Mingwei Fu, Zheshu Song, Zhikai Zhou, Shen Huang, Shidong Shang, Mengxiao Zhu

arXiv:2610.16260v1cs.AI

TL;DR

Acoustically ambiguous keywords can remain difficult for speech recognition, while existing benchmarks offer limited insight into how different kinds of visual evidence help. SCREENASR-Bench is a human-verified benchmark organized by evidence directness. Across three model families, visual context improves keyword recall, with larger gains for direct on-screen text than for indirect visual evidence.

  • Problem

    Existing audio-visual ASR benchmarks do not distinguish levels of visual evidence, leaving unclear which types models can exploit for acoustically ambiguous keywords.

    Source: §1 · p. 1

  • Method

    SCREENASR-Bench is a human-verified benchmark that groups challenging keywords into ETE, EVE, and ISE according to how directly visual context supports them.

    Source: §1 · p. 1, §2.2 · p. 3

  • Results

    Visual context consistently improves keyword recall across ASR, LALM, and Omni models, with larger gains for ETE than for EVE and ISE.

    Source: §4 · p. 4

  • Takeaways & Limitations

    The benchmark provides a foundation for research on visually grounded speech recognition.

    Source: §1 · p. 2, §4 · p. 4

Abstract

from arXiv · show

Current ASR models perform well on general benchmarks but still struggle with acoustically ambiguous words. Visual context can aid recognition in screen-based scenarios, yet existing benchmarks provide limited insight into the benefits of different types of visual evidence. To address this gap, we introduce ScreenASR-Bench , a tiered benchmark with three progressively challenging levels: Explicit Textual Evidence (ETE), based on direct on-screen text; Explicit Visual Evidence (EVE), using visible content rather than on-screen text; and Implicit Semantic Evidence (ISE), requiring scene-level inference. Extensive experiments on ASR models, Large Audio-Language Models (LALMs) and Omni models show that incorporating visual context substantially improves recall over the audio-only setting across all three model families. The gains are largest for ETE and smallest for ISE, indicating that models leverage direct visual evidence more effectively than indirect semantic cues. To facilitate future research on visually grounded speech recognition, we publicly release ScreenASR-Bench.

1. INTRODUCTION

Acoustically ambiguous keywords can remain difficult even when most of an utterance is transcribed correctly, and existing benchmarks do not distinguish visual cues by how directly they indicate a keyword. SCREENASR-BENCH evaluates three evidence levels and speech-model families to examine how visual context is used.

1. INTRODUCTION

  • Localized acoustic errors can alter utterance meaning while contributing little to aggregate WER, motivating focus on ambiguous target keywords.The paper asks how effectively current speech models exploit different levels of visual information for these keywords.Source: §1 · p. 1
  • Existing audio-visual benchmarks treat cases equally across visual-evidence types, leaving unclear which kinds of evidence models can exploit.Prior work also generally treats visual context as a whole rather than distinguishing cues by their directness.Source: §1 · p. 1
  • SCREENASR-BENCH categorizes cases as ETE, EVE, or ISE, from readable text to visible referents to indirect scene-level semantics.The semantic distance between the keyword and its visual evidence increases across these levels.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The benchmark systematically evaluates ASR, LALM, and Omni models on their use of visual context across all three evidence categories.Source: §1 · p. 2
  • The authors publicly release SCREENASR-BENCH to support reproducible research and further advances in multimodal ASR.Source: §1 · p. 2

2. SCREENASR-BENCH

SCREENASR-BENCH uses a three-stage construction pipeline that combines automatic screening with expert verification. Its three evidence levels distinguish how directly a keyframe supports a target keyword, and its 2,002 cases cover all three levels.

2.1. Dataset Construction Pipeline

  • A three-stage pipeline collects video/audio candidates, mines content-keyword ASR disagreements, uses a VLM to select and classify keyframes, then retains cases after expert verification.References are normalized; experts check transcription accuracy, audio difficulty, visual relevance, and evidence labels. Synthesized keyframes account for about 10% of the final benchmark.Source: §2.1 · p. 2

2.2. Visual Evidence Levels

  • ETE uses readable keyword text, EVE uses a directly visible referent, and ISE requires inference from scene-level semantics.Increasing semantic distance makes the visual evidence less direct; ISE therefore requires scene understanding and cross-modal reasoning.Source: §2.2 · p. 3

2.3. Dataset analysis

  • 2,002 cases comprise 39.31% ETE, 36.11% EVE, and 24.58% ISE; 91.82% of audio segments are shorter than 20 seconds.ISE is least frequent because implicit semantic evidence is harder to collect, though all three levels receive substantial coverage.Source: §2.3 · p. 3

3. EXPERIMENTS

Experiments compare audio-only and visual-context conditions across ASR, LALM, and omni models, using keyword recall alongside WER. Visual context improves recall across families, with larger gains for more direct evidence.

3.1. Experimental Setup

  • Models are tested with audio alone and, where supported, visual context; ASR systems and LALMs receive keyframe captions, while omni models receive captions and images.The evaluated systems span ASR, LALM, and omni model families.Source: §3.1 · pp. 3–4

3.2. Evaluation Metrics

  • WER measures overall transcription accuracy, while keyword Recall measures recognition of annotated keywords; NVG quantifies improvement from visual context over audio-only recall.NVG is 0 for no improvement, 1 when audio-visual recall is perfect, and negative when visual context degrades keyword recognition.Source: §3.2 · p. 4

3.3. Main Results

  • Visual context raises average recall from 48.18% to 62.20% for ASR models, 47.24% to 64.33% for LALMs, and 61.98% to 77.24% for omni models.The corresponding average WER decreases from 6.02% to 5.19%, from 8.19% to 6.65%, and from 6.47% to 4.74%, respectively.Source: §3.3 · p. 4
  • ASR recall gains fall from 26.53% for ETE to 6.95% for EVE and 4.39% for ISE; LALMs and omni models follow the same ranking.Mean NVG follows a similar level-wise trend across all three model families.Source: Table 2 · p. 4, §3.3 · p. 4
  • For Qwen3-Omni, captions and images yield 74.09% versus 74.89% recall, but captions lower WER (4.01% versus 5.49%), a pattern shared by other omni models.The authors suggest captions provide compact information closely aligned with speech, while raw images may introduce irrelevant visual information.Source: §3.3 · p. 4

4. CONCLUSION

SCREENASR-BENCH is a human-verified, tiered benchmark for keyword recognition with visual context. Experiments find recall improvements across model families, with larger gains from direct than indirect visual evidence.

  • SCREENASR-BENCH is a human-verified, tiered benchmark for evaluating how speech models use visual context to recognize challenging keywords.It covers Explicit Textual Evidence (ETE), Explicit Visual Evidence (EVE), and Implicit Semantic Evidence (ISE).Source: §4 · p. 4
  • Visual context consistently improves keyword recall across ASR, LALM, and omni models, with larger gains for ETE than for EVE or ISE.The authors report that current models exploit on-screen text more effectively than indirect visual evidence.Source: §4 · p. 4
  • The results highlight the need for stronger visual grounding and reasoning in speech models.Source: §4 · p. 4

Related papers

Loading 2610.16260v1…