PaperlayerView exact revision on arXiv

Source-linked AI summary

Mind The Gap (MTG-AD): Dataset and Fine-grained Evaluation for End-to-End Inline Audio Descriptions

Subhashini Venugopalan, Jenny Tan, Taylor Roper, Jimmy Tobin, Anton Kast, Alicia Martin, Sam Sepah, Amy Pavel

arXiv:2610.39955v1cs.CVcs.AIcs.HC

TL;DR

End-to-end inline audio description for user-generated videos remains under-explored, despite its importance for BLV access. The paper introduces a professionally co-authored benchmark and reference-free evaluation framework that tests both content and rendered timing. Frontier MLLMs approach human visual descriptiveness, but gaps remain in temporal precision and narrative pacing.

  • Problem

    Prior evaluation does not capture the full inline task of generating descriptions, rendering them with TTS, and testing their fit within natural silences.

    Source: §1 · p. 2

  • Method

    The paper pairs a professional inline-AD dataset with an end-to-end, reference-free framework using TTS temporal metrics and multimodal LLM judges.

    Source: §1 · p. 2, §4 · p. 5, §4.2 · p. 6

  • Results

    No model leads on every metric: newer-generation omni-modal models raise content prioritization from 0.76 to 0.79–0.94, while human AD remains better paced.

    Source: §6 · p. 10

  • Takeaways & Limitations

    Select models first for temporal feasibility using overlap and coverage, then optimize prioritization, non-redundancy, and reference-free clarity and descriptiveness.

    Source: §6 · p. 10

    Temporal metrics use standard TTS at a fixed 1.2× speaking rate, so absolute overlap and coverage values depend on that rate.

    Source: §6 · p. 10

Abstract

from arXiv · show

Audio descriptions (AD) are essential for making visual media accessible to blind and low-vision (BLV) users. While Multimodal Large Language Models (MLLMs) offer a scalable solution for generating audio descriptions on-demand, their performance on ``inline'' audio descriptions -- which must fit within existing silences in a video -- remains under-explored, particularly for diverse user-generated content (UGC). We present a comprehensive investigation into MLLM-generated inline audio descriptions. First, we introduce MTG-AD, a professionally annotated dataset of 36 hours of videos, each averaging $\sim$5 mins. and 8k+ descriptions in total, co-created by paired blind and sighted AD professionals to provide a high-quality benchmark for this task. Second, we propose MTG-AD-Eval, an evaluation framework based on audio description expert guidelines that provides fine-grained, reference-free actionable metrics for model improvement. Crucially, our framework evaluates the end-to-end task: it renders the generated scripts into audio tracks to assess their fit within the original video's natural silences, minimizing disruptive overlaps. Our analysis across frontier MLLMs reveals that while models accurately describe visual events and substantially reduce speech overlaps, gaps persist in narrative pacing and flow. Finally, we empirically validate that our automated and reference-free metrics correlate significantly with human quality ratings from sighted raters and 7 BLV participants, finding that while users narrowly prefer human-authored audio descriptions, they find MLLM-generated audio descriptions highly beneficial for comprehension. We provide actionable system design guidance and release MTG-AD at https://hf.co/datasets/google/mind-the-gap-inline-ad and MTG-AD-Eval at https://github.com/google/mtg-ad.

1 Introduction

Inline audio descriptions must fit natural pauses while adding coherent visual information without repeating or interrupting existing audio. This work addresses the lack of end-to-end evaluation for MLLM-generated inline descriptions on user-generated videos with a professional benchmark and reference-free evaluation framework.

1 Introduction

  • Inline AD must fit natural pauses and selectively narrate visual events to build coherent stories without repeating or interrupting existing audio.For BLV viewers, this matters in user-generated videos where creators rarely verbalize essential visual details [18].Source: §1 · p. 1
  • Prior evaluations miss sequence coherence, spoken duration, dead air, and speech overlap, while user-generated videos often lack expert annotations.Earlier work evaluates trimmed clips against single references, short-clip ratings, or coarse LLM scores [35] [15].Source: §1 · p. 2
  • MTG-AD provides professionally crafted inline AD for 433 YouTube videos, while MTG-AD-Eval assesses generated descriptions end-to-end with reference-free metrics validated against human judgments.The evaluation includes sighted raters and 7 BLV participants, and identifies remaining gaps in MLLM narrative pacing [12].Source: §1 · p. 2

2 Related Work

Prior audio-description datasets and evaluations often focus on scripted entertainment, isolated clips, or text-to-reference comparisons. MTG-AD addresses the user-generated-video setting with full-length, professionally co-authored descriptions and end-to-end inline evaluation.

2 Related Work

  • Movie and TV benchmarks use scripted entertainment, while user-generated videos vary in intent, camera work, editing pace, on-screen text, and speech density.Prior movie AD benchmarks commonly evaluate pre-trimmed 2–20 s clips around known description timestamps [8] [29] [35].Source: §2 · p. 2
  • MTG-AD contributes 36 hours of full-length user-generated videos co-annotated by sighted and blind AD professionals.Its dataset statistics cover 433 videos and show vlogs as the largest category alongside travel, nature, and education.Source: §2 · p. 3, Figure 2 · p. 3
  • Reference-based text metrics can penalize valid alternatives, and script-only evaluation does not test fit within natural pauses or speech overlap.Prior metrics include BLEU and BERTScore [37], alongside domain-specific measures such as MNScore [35], CRITIC, and LLM-AD-Eval.Source: §2 · p. 3
  • Earlier work had not evaluated end-to-end inline AD on full-length user-generated videos against professionally co-authored AD while validating reference-free metrics with listeners.Short-clip studies had examined guideline-prompted MLLM descriptions and LLM-judge alignment with BLV judgments [15] [33].Source: §2 · p. 3

3 Mind the Gap (MTG-AD) Dataset

MTG-AD is a professionally co-authored benchmark of full-length user-generated videos, and the paper studies model-generated descriptions with increasingly explicit timing guidance. Its construction and prompting setup target the joint challenge of selecting visual content and fitting speech into natural pauses.

3 Mind the Gap (MTG-AD) Dataset

  • MTG-AD benchmarks inline AD on 433 English YouTube videos totaling 35.9 hours, with validation and test splits, paired-professional descriptions, ASR transcripts, and visual event lists.The test release includes scripts and metric scores for 12 model–prompt configurations spanning four prompting strategies and nine frontier models.Source: §3 · p. 3

3.1 Video Selection and Filtering

  • The team retained 433 unique videos after filtering candidates for temporal viability and topical diversity.From 557 candidates, 92 were excluded as non-viable and 27 skipped for redundancy; borderline videos could receive partial descriptions where pauses permitted.Source: §3.1 · p. 3

3.2 Professionally Co-Authored AD

  • Sighted writers drafted concise timed descriptions, adjusted them against the soundtrack, and blind partners reviewed full playback for clarity and comprehension.The resulting dataset contains 8,126 AD segments, averaging 18.8 segments per video.Source: §3.2 · p. 4

3.3 Eliciting Model-Generated AD on Untrimmed Videos

  • The four prompting strategies progress from minimal instructions to expert guidelines, timestamped ASR, and explicit per-gap duration and word-budget planning.For Gemini-2.5 Flash, guideline prompts improve visual prioritization and on-screen text recall over Simple AD; Gd. + Timing ASR best balances content and timing.Source: §3.3 · p. 4, Figure 3 · p. 4

4 MTG-AD-Eval: Evaluation Framework

MTG-AD-Eval translates expert audio-description guidelines into fine-grained metrics for content, timing, audio fit, and holistic script quality. It evaluates rendered descriptions against video and speech information, with most metrics usable without human-written references.

4 MTG-AD-Eval: Evaluation Framework

  • The framework distills expert guidelines and BLV audience feedback into automatic content and temporal metrics, alongside VideoA11y writing-quality rubrics.The selected guidelines target prioritization, redundancy, on-screen text, timeliness, overlaps, and audio coverage.Source: §4 · pp. 4–5
  • Metrics score whole-video descriptions using TTS-rendered intervals and timestamped ASR speech plus dense visual information, avoiding human references except for VideoA11y-Ref.Descriptions are synthesized at a fixed 1.2× speaking rate; Quality-RF judges scripts against the video and transcript.Source: §4 · p. 5

4.1 Content Metrics: What the AD Says

  • On-Screen Text recall measures whether words shown in the video appear in the AD or transcript, while prioritization and redundancy assess salient-content precision and repeated audible information.Videos without on-screen text are excluded from the On-Screen Text score.Source: §4.1 · p. 5

4.2 Temporal Metrics: When and How Much the AD Speaks

  • Temporal metrics measure event-to-description timing, AD–speech overlap, AD duration relative to available silence, and the mean remaining gap when neither speech nor AD plays.Audio Coverage above 1 flags over-generation; values somewhat below 1 are desired, and the ratio is not clipped.Source: §4.2 · p. 6

4.3 Holistic Script Quality

  • VideoA11y-Ref compares generated scripts with human AD, whereas Quality-RF removes that reference and judges the video-grounded script on the same four writing dimensions.The reference-based Accurate score effectively measures reference alignment, penalizing correct details omitted by the human describer.Source: §4.3 · p. 6

5 Experiments and Results

Across prompting strategies and frontier MLLMs, timing and content metrics expose distinct trade-offs, while judge calibration and human studies qualify how scores should be interpreted. Sighted and BLV ratings support the usefulness of the evaluation metrics, with some differences between human- and model-authored descriptions.

5.1 Metric-based Evaluations

  • 12.77 s of speech overlap with Guideline + Timing ASR, compared with 21.34 s for Guideline + ASR and 59.19 s for Guideline.Guideline + Timing ASR preserves Prioritization at 0.76; Human AD has 7.51 s overlap and 4.19 s Avg. Silence.Source: §5.1 · p. 7
  • Qwen-3.5 and Gemini models score 0.76–0.94 on Prioritization, versus 0.51–0.59 for other models, but no model leads on every metric.Gemini-3.5 Flash has 0.26 s overlap and 4.85 s Avg. Silence with Coverage 0.45; Qwen models reach Coverage 0.72–0.94 but incur 30.45–52.14 s of overlaps.Source: §5.1 · p. 7
  • Scores of 1.16–1.63 on Reference Alignment (Accurate) under the Gemini-2.5 Flash judge partly reflect penalties for valid details absent from the human reference.† models score higher, but their Gemini-3.5 Flash judge differs, so the score jump may partly reflect judge changes.Source: §5.1 · p. 7, Figure 5 · p. 7

5.2 LLM as Judge and Generalization

  • All judge pairs correlate positively (r = 0.38–0.74), with GPT-4.1 mini and Qwen-2-VL-Max agreeing most; judge leniency differs, so comparisons should use one judge.Figure 6 summarizes inter-judge Pearson correlation across 101 test videos and deviations from the judges’ global average scores.Source: Figure 6 · p. 7, §5.2 · p. 8
  • On YouDescribe, Premium human AD trends ahead of Standard volunteer AD on Timeliness (2.16 s vs. 2.85 s) and Prioritization (0.82 vs. 0.80), with overlapping confidence intervals.The evaluation suite was applied to external videos whose available silence gaps were significantly longer than in MTG-AD.Source: §5.2 · p. 8, Figure 7 · p. 8

5.3 Sighted and BLV Participant Studies & Metric Validation

  • Eight sighted raters’ agreement across metrics was Krippendorff’s α = 0.654, and Gemini-2.5 Flash with Gd. + Timing ASR approached human ground truth in Accuracy and Timeliness but lagged on audio overlap issues.Figure 8 reports mean Likert scores across five key dimensions; Gemini-2.5 Flash approaches Ground Truth on script Quality as well.Source: §5.3 · p. 8, Figure 8 · p. 8
  • Seven BLV participants rated both AutoAD and professional Human AD consistently high; 53.6% marginally preferred Human AD, with significant differences only in Satisfaction and perceived Accuracy.Preferences varied by genre: several participants preferred denser AutoAD detail for action or travel clips and concise human pacing for dialogue-heavy vlogs.Source: §5.3 · p. 9, Figure 9 · p. 9
  • For sighted raters, Audio Overlaps and Avg. Silence significantly tracked matching ratings; for BLV listeners, No Redundancy and VideoA11y-Ref Clear were significant predictors.Quality-RF changed the BLV Accuracy correlation from ρ = −0.264 for VideoA11y-Ref Accurate to ρ = +0.244, and correlated with Descriptive (ρ = +0.583, p = 0.001) and Clear (ρ = +0.415, p = 0.028). BLV listeners did not hear speech collisions because playback cut AD off when speech began.Source: §5.3 · pp. 9–10, Table 3 · p. 9

6 Conclusion and Broader Impact

Frontier MLLMs approach human audio descriptions in visual descriptiveness, but gaps remain in temporal precision, silence coverage, and pacing. The paper reports benefits for BLV comprehension and offers system-selection guidance, while identifying limits in its metrics, study size, and fixed-rate TTS.

6 Conclusion and Broader Impact

  • Gemini-3.5 Flash cuts speech overlaps to 0.26 s, while newer omni-modal models raise content prioritization from 0.76 to 0.79–0.94; no model leads every metric.The models trade overlap against silence coverage, and human AD remains better paced.Source: §6 · p. 10
  • BLV participants found MLLM-generated AD highly beneficial for comprehension, although human-crafted AD remained the gold standard for narrative flow.The validated reference-free metrics correlated substantially with sighted and BLV human judgments.Source: §6 · p. 10
  • System selection should first filter for temporal feasibility using Audio Overlaps (G7) and Audio Coverage (G8) ≤1, then optimize content and narrative metrics.This still permits undescribed gaps; subsequent optimization maximizes Prioritization (G1.1), No Redundancy (G1.2), and Quality-RF.Source: §6 · p. 10
  • The automated metrics cover six BLV-pilot guidelines, while G1.1 and G2 depend on MLLM-extracted events and judge calibration.The BLV study included 7 participants and 28 full-length video evaluations; temporal metrics use standard TTS at a fixed 1.2× speaking rate.Source: §6 · p. 10
  • MTG-AD and MTG-AD-Eval are released with human and model ADs, ASR transcripts, and pre-extracted visual events to support reuse and reproducibility.The framework and dataset could support auditing and generation for tutorials, educational content, and vlogs, but generated AD can mislead through hallucinations and should be labeled as AI-generated.Source: §6 · p. 10

A Synthesized Suite of metrics from Expert Guidelines

The evaluation suite translates expert audio-description guidelines into metrics for content, timeliness, language, and audio fit. The initial metrics prioritize issues identified by BLV participants in a pilot study.

A Synthesized Suite of metrics from Expert Guidelines

  • Content metrics cover prioritizing necessary visual information, avoiding redundancy, narrating on-screen text, and describing visual style only when useful.The suite also evaluates timeliness, accuracy, and objectivity of descriptions.Source: A Synthesized Suite of metrics from… · p. 14
  • Language guidelines assess active voice, audience appropriateness, consistency with the video’s speech, and vocabulary and tone.Appropriateness includes matching the AD language to the video's language.Source: A Synthesized Suite of metrics from… · p. 14
  • Audio guidelines assess overlap avoidance, coverage of most—but not necessarily all—video time, pronunciation, and suitable speech style, speed, and prosody.The appropriate speech style can depend on the video's content.Source: A Synthesized Suite of metrics from… · p. 14
  • The main experiments use G1.1–G1.3, G2, G7, and G8, selected after BLV pilot participants reviewed descriptions generated from a basic prompt.The pilot surfaced issues that most negatively affected participants’ experience.Source: A Synthesized Suite of metrics from… · p. 14

B Full Results on the Proposed Metrics

The appendix presents detailed prompt and model comparisons, inter-annotator statistics, and BLV rating distributions. The supplied passages describe the comparisons' scope but do not provide their underlying score values.

B Full Results on the Proposed Metrics

C Inter Annotator Agreement and Sighted Rater Prompt Comparisons

D Full Distribution of Scores and Qualitative Insights from the BLV User Study

  • The BLV subsection presents score distributions across eight dimensions and summarizes qualitative participant feedback.The supplied passage does not state the distribution results or feedback findings.Source: D Full Distribution of Scores and… · p. 16

D.1 BLV Qualitative Feedback and Individual Preference Variance

BLV participants’ preferences varied by individual and genre: human AD was favored overall for flow, while AutoAD was valued on visually dense videos. Ratings were generally high for both versions, with some lower marks for automated AD.

D.1 BLV Qualitative Feedback and Individual Preference Variance

  • Human AD was preferred in 53.6% of paired evaluations, while AutoAD was preferred in 46.4%.Participants valued AutoAD’s exhaustive enumeration on visually dense travel, nature, and instructional videos; human AD was praised for smooth flow, concise phrasing, and integration around dialogue pauses.Source: §D.1 · p. 16
  • Participants sometimes penalized AutoAD for redundancy with sound effects or abrupt truncation when speech began.The study involved 7 BLV participants and 28 paired video evaluations, with substantial individual and genre-dependent preference variance.Source: §D.1 · p. 16
  • BLV participants rated both human AD and AutoAD consistently high, at 4 or 5 across all eight dimensions.Some automated AD videos received lower ratings for audio quality, overall quality, and satisfaction than the same videos with human AD.Source: Figure 11 · p. 17
  • When choosing between versions, participants marginally preferred videos with human-generated AD.The blind preference comparison used AutoAD generated by Gemini-2.5 Flash with Gd. + Timing ASR.Source: Figure 12 · p. 17

E Extended Empirical Metric Validation and Reference-Free Evaluation

The extended validation compares automated metrics with sighted and BLV judgments and tests reference-free inputs for Quality-RF. The section also specifies the sighted raters’ rubric and notes that temporal metrics depend on the chosen TTS speaking rate.

E Extended Empirical Metric Validation and Reference-Free Evaluation

  • Video + ASR achieved the strongest overall Quality-RF correlations: ρ = +0.583 on Descriptive, ρ = +0.415 on Clear, and ρ = +0.244 on Accurate.Text-only dense visual info + ASR had the highest standalone Accuracy correlation (ρ = +0.289) when raw video frames could not be passed to the judge.Source: E Extended Empirical Metric Validation… · p. 17
  • Replacing human reference text with raw video and ASR reversed the negative Accuracy correlation and strengthened alignment across all four rubric dimensions.The section supplies full correlation tables for sighted raters (n = 42 videos), BLV participants (n = 28 video evaluations), and the reference-free comparison.Source: E Extended Empirical Metric Validation… · p. 17, Table 10 · p. 18
  • The temporal metrics G7 and G8 use TTS at a fixed 1.2× rate, so their absolute values depend on that rate; other rates were not studied.Faster or adaptive per-gap rates are identified as a promising direction for fitting more descriptions into available silences.Source: E Extended Empirical Metric Validation… · p. 18

F Questions for Sighted Raters

  • The rubric’s descriptiveness criterion combines vivid detail with concise narrative flow, while objectivity excludes personal opinions or assumptions.Raters also considered whether descriptions were timely, free of audio issues such as overlapping speech, and whether long silences remained.Source: F Questions for Sighted Raters · p. 20

G Examples showing Inline Audio Description Constraints

The examples contrast professional and model-generated inline descriptions, highlighting differences in detail, perspective, narrative flow, and event coverage. The prompts range from basic instructions to transcript-aware timing rules that constrain what can be narrated in each silence.

G Examples showing Inline Audio Description Constraints

  • The examples show human annotators providing more detail despite tight timing constraints and describing video perspective alongside vivid narration.These examples illustrate differences from model-generated descriptions in detail and narration style.Source: Figure 13 · p. 21, Figure 14 · p. 22

H Examples with different prompts

  • In the prompt examples, professional descriptions follow a woman’s story rather than listing actions, while a model-generated description misses a truck rapidly braking for an elephant.The examples compare generated and professionally annotated AD for video segments shown with manually selected frames.Source: Figure 15 · p. 23

I Prompts for AD Generation

  • The guideline prompt asks for concise, factual descriptions of essential visual information, including on-screen text, and assumes speech rates of 1–3 words per second.The transcript-aware prompt supplies timestamped speech boundaries and asks that descriptions fit between dialogue.Source: I Prompts for AD Generation · pp. 24–25
  • The basic timing prompt requires descriptions only in speech gaps, caps each at three words per available second, and says to skip visuals that cannot fit.It also limits each timestamped description to three sentences and asks for more descriptions when a silent video has no speech.Source: I Prompts for AD Generation · p. 27

J Prompts for AD Evaluation Metrics

The evaluation prompts cover event prioritization, redundancy, timeliness, and reference-based and reference-free quality scoring. The reference-free Quality-RF prompt evaluates descriptions against raw video and ASR without human ground-truth AD.

J Prompts for AD Evaluation Metrics

  • Separate prompts evaluate prioritization of important events, redundancy with the original transcript, timeliness against pre-extracted visual information, and reference-based VideoA11y-Ref metrics.These prompts target distinct evaluation questions rather than a single overall score.Source: Figure 21 · p. 28, Figure 23 · p. 29, Figure 24 · p. 29
  • The initial On-Screen Text metric prompt used an LLM judge to count unsaid words, but reported G1.3 results instead use programmatic word-level recall.The reported measure is the fraction of on-screen-text words in dense visual information that appear anywhere in the AD or ASR transcript; no LLM judge is involved.Source: Figure 22 · p. 28
  • The reference-free Quality-RF prompt evaluates AD against raw video and ASR without requiring human ground-truth AD.The prompt is identified as the Video + ASR configuration.Source: J Prompts for AD Evaluation Metrics · p. 30, Figure 26 · p. 31
Loading 2610.39955v1…