PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniCapBench: A Deep-Structured Evaluation Framework for Fine-Grained Audio-Visual Captioning

Zhongyu Yang, Jiale Tao, Ruitao Chen, Zuhao Yang, Yingfang Yuan, Xueliang Zhao, Auden, Kai Wang, Shuai Shao, Biao Wang, Steve Yves, Qinglin Lu

arXiv:2610.12458v1cs.CV

TL;DR

Existing audio–visual caption evaluation struggles to combine broad coverage, localized diagnosis, and reliable scoring. OmniCapBench evaluates atomic Reference, Shot, and Event units with deterministic structural checks and localized semantic comparisons; evaluated models still show weaknesses in cross-modal association and instability in global judging.

  • Problem

    Existing audio–visual caption benchmarks lack a jointly designed evaluation unit and scoring operator for fine-grained, reliable diagnosis.

    Source: §1 · p. 2

  • Method

    OmniCapBench represents videos with atomic Reference, Shot, and Event units, combining deterministic structural checks with bounded local semantic matching.

    Source: §1 · pp. 2–3, §3.2 · p. 6

  • Results

    Holistic scores mask localized errors and global, unlocalized LLM judging induces unstable conclusions; Gemini 3.1-Pro scores 63.32% Event F1 and 51.46% Event-Shot Association (EVSA F1).

    Source: §1 · p. 3, §4.2 · p. 8

  • Takeaways & Limitations

    Deep-structured evaluation preserves comprehensive video coverage while retaining fine-grained detail for diagnosis.

    Source: §5 · p. 10

    Evaluation is limited to facts in the annotated reference system, so valid peripheral details outside annotation scope go uncredited.

    Source: C Limitations and Broader Impact · p. 37

Abstract

from arXiv · show

Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that expose their capability limits. Audio--visual captioning is an ideal diagnostic task, yet current benchmarks face a coupled trade-off: whole-caption scores provide coverage without localization, local probes provide localization without coverage, and unconstrained LLM judges introduce instability. We introduce OmniCapBench (Omni-Video Caption Benchmark), a benchmark that reframes audio--visual caption evaluation as a deep-structured diagnostic framework. OmniCapBench shifts the prediction target from free-form text to sets of atomic, verifiable evaluation units across three tracks: entity references, visual shots, and audio events, enabling reliable scoring with deterministic constraint checks and localized LLM-based semantic comparisons. With 786 densely annotated videos, OmniCapBench effectively distinguishes MLLM perception errors, including temporal grounding failures, identity drift, cross-modal misalignment, and hallucinated descriptions. Evaluating frontier MLLMs reveals strong local perception but weak long-horizon audio--visual reasoning, particularly in identity drift and cross-modal misalignment, providing a fine-grained roadmap for omnimodal development.

1 Introduction

Existing audio–visual caption evaluation struggles to combine broad coverage, localized diagnosis, and reliable scoring. OmniCapBench uses atomic evaluation units and decoupled scoring to support fine-grained analysis, while its empirical audit finds instability in global, unlocalized LLM judging.

1 Introduction

  • Existing protocols trade whole-caption coverage for sparse localized probes, while post-hoc extraction and unconstrained LLM judging complicate reliable scoring.Whole-caption scores can hide local errors, whereas probe-based units offer localization but lack coverage.Source: §1 · p. 2
  • OmniCapBench decomposes captions into atomic, verifiable units and separates deterministic structural checks from localized LLM semantic comparisons.Its tracks cover references, shots, and events, supporting checks of temporal grounding, identity tracking, and cross-modal association.Source: §1 · pp. 2–3
  • The empirical audit finds that global text scores mask localized errors and global, unlocalized LLM scoring can induce unstable conclusions.The audit compares holistic evaluation across models, metrics, and generation formats, and contrasts weak and strong LLM judges.Source: §1 · p. 3
  • The benchmark contains 786 densely annotated videos, with 5,818 entities, 6,537 audio events, and 11,419 visual shots across its native tracks.The proposed paradigm isolates semantic content, temporal grounding, identity tracking, and cross-modal association.Source: §1 · p. 3

2 Related Work

Prior video and caption evaluations often lack diagnostic resolution, while caption protocols face trade-offs between coverage and localization. OmniCapBench combines deep-structured units with separated structural and semantic scoring.

2 Related Work

  • Broad video benchmarks often reduce complex behaviors to coarse text-level scores that lack resolution for pinpointing localized audio–visual failures.Holistic caption scoring can mask temporal or relational errors, while QA and temporal probes localize errors but have sparse coverage.Source: §2 · p. 3
  • Global LLM judges entangle parsing, reference resolution, and temporal tracking, conflating factual hallucinations with stylistic variation and making scores unreliable.Other protocols also rely on fragile post-hoc extraction of script items from free-form text.Source: §2 · p. 3
  • OmniCapBench targets the coverage-localization trade-off with native atomic units, deterministic structural probes, and bounded localized LLM semantic checks.Its construction filters videos for audio–visual complexity, iteratively generates References, Events, and Shots, then applies human audits.Source: §2 · pp. 3–4, Figure 2 · p. 4

3 OmniCapBench Benchmark

OmniCapBench builds a structured reference system for fine-grained, reliable evaluation of omnimodal models. It evaluates native units through structural validation, matching, and localized semantic scoring.

3.1 Data Construction and Native Track Generation

  • The reference system adopts MTSS’s factorized streams and refines them into persistent References, timestamped Events, and visual Shots that anchor entities and sounds to frames.Visual shots are decomposed into finer subshots, and point timestamps are replaced with continuous time ranges.Source: §3.1 · p. 4
  • Construction filters videos for audio–visual complexity, iteratively builds and validates track structure before refining descriptions, then audits the finalized units programmatically and with human review.The dataset comprises 786 videos; its category-distribution figure uses segment area to represent video counts across two category levels.Source: §3.1 · p. 5, Table 2 · p. 5, Figure 3 · p. 5

3 OmniCapBench Benchmark

  • OmniCapBench derives diagnostic tasks from a native reference system rather than unconstrained text generation, separating audio–visual comprehension from language style.Its benchmark is intended to support high-granularity and reliable evaluation.Source: §3 · p. 4, §3.2 · p. 5

3.2 Evaluation Goals and Task Suite

  • Native Unit Generation requires structured Reference, Shot, and Event predictions; structural checks remove invalid IDs, time spans, and incompatible links before matching valid units.Shots and events are matched by temporal intersection-over-union, while identity references and visual subshots use a bounded LLM-assisted matcher on local candidate pairs.Source: §3.2 · pp. 5–6
  • Localized semantic scoring compares only structurally aligned unit pairs, while bidirectional Precision and Recall separately penalize hallucinations and omissions.Reported structural metrics span visual, audio, and audio–visual domains; semantic scores are reported separately for aligned units.Source: §3.2 · pp. 6–7, Table 3 · p. 6, Table 4 · p. 7

4 Experiments

OmniCapBench evaluates audio–visual captioning through structural and localized semantic checks, revealing model weaknesses that holistic text scores obscure. Experiments also find that global scores can hide errors and vary sharply with the LLM judge.

4.1 Experimental Setup and Metrics

  • The experiments evaluate representative proprietary, open-source, and specialized captioning models with rule-based metrics and separate semantic scores on structurally aligned units.Specialized captioning models are excluded from the main evaluation because their free-form-text fine-tuning leads to poor structured-generation instruction following.Source: §4.1 · p. 7

4.2 Benchmarking Omnimodal Model Capabilities

  • Gemini 2.5-Pro’s Cross-Shot Coreference Consistency falls to 37.81%, while open-source models fall below 12%, despite 84.80% Ref Subject F1 on short videos.The comparison highlights a gap between basic entity recognition and tracking identity across cuts.Source: §4.2 · pp. 7–8
  • Gemini 3.1-Pro drops from 63.32% Event F1 to 51.46% EVSA F1, while open-source models stay below 20% EVSA F1; Seed2.0 records 4.54% Event F1 and 3.52% EVSA F1.Seed2.0 nevertheless reaches 91.80% speaker attribution.Source: §4.2 · p. 8
  • Localized semantic checks expose a precision–recall trade-off: Gemini 3.1-Pro scores 67.07% precision and 43.78% recall on subjects, while MiniCPM-o-2.6 reaches 76.32% Scene precision but 20.39% recall.Other models raise recall by hallucinating unverified attributes.Source: §4.2 · p. 8
  • Frontier-model failures concentrate in action hallucination (28.5%), A-V misalignment (23.7%), and audio hallucination (17.9%), while open-source errors span basic perception and cross-modal binding.Format errors account for 1.3% of Gemini 3.1-Pro failures; Figure 4 contrasts these failure profiles.Source: §4.2 · p. 8, Figure 4 · p. 8

4.3 Auditing the Legacy Evaluation Paradigm

  • Scores for identical outputs rise from 71.24 with a weak judge to 93.92 with a strong judge on holistic and QA baselines; OmniCapBench instead confines judges to bounded semantic checks on aligned units.The judges compared were Qwen3.6-27B, GPT-4o, and Gemini-2.5-Pro.Source: §4.3 · p. 9
  • 74.2% versus 25.8% under structural constraints separates stronger from weaker models that appeared near-equal (~50% vs. ~50%) on text metrics, aligning with human audit.The comparison uses hard-to-distinguish subsets of 200 videos.Source: Figure 4 · p. 8, §4.3 · p. 9

5 Conclusion

OmniCapBench recasts audio–visual caption evaluation as verifiable atomic units with deterministic structural checks and localized semantic scoring. Its analyses find that holistic text scores mask local errors and unconstrained LLM judges produce ranking instability.

5 Conclusion

  • OmniCapBench uses atomic Reference, Shot, and Event units to provide broad video coverage while preserving fine-grained detail and separating structural verification from local semantic checks.The conclusion frames verifiable atomic units as the basis for actionable diagnosis.Source: §5 · p. 10
  • The conclusion reports that global text scores can mask localized errors and reward structurally inconsistent outputs, while unconstrained LLM judges introduce ranking instability.Source: §5 · p. 10

Overview of Appendix

  • The appendix collects construction details, additional results, limitations and broader impact, ethical considerations, dataset licenses, and executable instruction templates.It is presented as supplementary material for reproducibility and transparency.Source: Overview of Appendix · p. 17

A.1.1 Benchmark Construction Details

OmniCapBench construction combines curated videos, structured multi-model annotation, and human quality assurance. Its generation and evaluation protocols enforce structural validity and temporal grounding across a 786-video testbed.

A.1.1 Benchmark Construction Details

  • Candidate videos are filtered for technical quality, available audio, duration buckets, and minimum structural density before a multi-role pipeline drafts, verifies, evaluates, refines, and selects annotations.Candidate selection requires enough shots, events, and characters per minute for deep-structured evaluation.Source: §A.1.1 · p. 17

A.1.2 Output Schema

  • Models must produce the prescribed references, events, and shots JSON schema; non-parseable outputs trigger a parse failure and zero scores on downstream metrics.Inference uses greedy decoding with a maximum output limit of 4096 tokens.Source: §A.1.2 · p. 18, Table A.3 · p. 18

A.1.3 Generation and Evaluation Protocols

  • Generation supports single-pass or progressive processing, while instructions enforce atomic visual actions, overlapping concurrent subshots, objective descriptions, and audio-grounded event reporting.The progressive strategy extracts persistent entities first, then generates events and shots in sliding temporal segments.Source: §A.1.3 · p. 19

A.1.4 Dataset and Domain Statistics

  • The benchmark contains 786 videos spanning multiple public datasets and durations up to five minutes, with statistics characterizing category coverage, temporal scale, and per-minute structural density.Figures A.1–A.4 describe source and category distributions, durations, unit counts, and density patterns.Source: §A.1.4 · pp. 19–20, Figure A.1 · p. 20, Figure A.2 · p. 21
  • Human reviewers iteratively refine annotations for structural integrity and frame-level temporal precision, escalating difficult cases to senior reviewers for adjudication.The review interface uses synchronized video-audio playback.Source: §A.1.4 · pp. 21–22

A.2.1 Experiments Compute Resources

  • Reference matching uses an LLM-assisted one-to-one comparison of identity descriptions, while event matching uses transcript WER or temporal IoU and shot matching uses temporal IoU.Experiments run on nodes with 8× NVIDIA H20 GPUs and use vLLM for local serving.Source: §A.2.1 · pp. 22–23
  • LLMs are restricted to bounded local semantic checks after structural alignment, separating semantic equivalence judgments from the matching and verification stage.Source: §A.2.1 · p. 23

A.2.2 Metric Definitions

OmniCapBench defines characters, sound events, and visual actions as verifiable atomic units whose structural interactions are checked before localized LLM scoring. Bidirectional matching addresses open-ended caption mismatches, while bounded scoring distinguishes annotated truths from unsupported extras.

A.2.2 Metric Definitions

  • Characters, sound events, and visual actions are treated as verifiable atomic units, with deterministic rules checking their interactions before localized LLM scoring.Source: §A.2.2 · p. 23
  • Granularity, attribute selection, and structural topology can differ between valid human and model annotations, making recall-only or precision-only scoring biased toward verbosity or genericity.OmniCapBench applies bidirectional matching and scoring to References and visual Subshots.Source: §A.2.2 · p. 23
  • Reference matching compares identity-detail descriptions one-to-one; dialogue events use normalized WER, non-dialogue events and shots use temporal IoU, and subshots match locally within aligned shots.The specified thresholds are 0.50 WER, 0.20 event tIoU, and 0.30 shot tIoU, with a 1.0-second expanded subshot window.Source: §A.2.2 · p. 24
  • The open-world scoring policy does not penalize correctly identified truths outside the annotation budget, while unsupported extras are logged diagnostically rather than deducted from main precision metrics.Source: Table A.6 · p. 24, §A.2.2 · p. 24

A.2.3 Full Model and Evaluation Settings

The evaluation settings distinguish native-capable models from captioning baselines and document reproducibility constraints. Deterministic structural checks and alignment procedures expose failures that holistic caption scores can conceal.

A.2.3 Full Model and Evaluation Settings

  • The study assesses 12 models; its main benchmark covers seven native-capable models, while five captioning baselines use a caption-posthoc route because their fine-tuning degrades schema-following.Local Qwen3-Omni models are capped at 192 frames at 2 fps, and videos over one minute use a two-pass strategy; Qwen2.5-Omni-based models are limited to videos of 60 seconds or less.Source: §A.2.3 · pp. 24–25, Table A.7 · p. 25
  • 65.2% of outputs failed Identity Persistence and 42.0% failed Temporal Alignment, despite high text-level scores.These structural failures indicate difficulty maintaining object permanence across shots and anchoring events to precise timeline segments.Source: §A.2.3 · p. 26

A.2.4 Consistency Checks

  • Deterministic predicates check valid reference IDs, temporal overlap and compatibility, consistent coreference, required evidence fields, and dangling links.Examples include rejecting event-shot links with no positive temporal overlap and identity assignments that split across sequential cuts.Source: §A.2.4 · pp. 26–27
  • Cross-reference integrity failed at 65.2% and Temporal compatibility at 42.0%, highlighting structural reasoning failures beyond isolated frame detection.The probes assess structurally applicable predictions for Gemini 3.1-Pro.Source: §A.2.4 · p. 27, Table B.1 · p. 28

A.2.5 Alignment Protocol

  • Canonicalization and temporal matching are rule-based, while reference alignment uses bounded LLM-assisted one-to-one matching; rule-based thresholds and tie-breaks are fixed before scoring.The pipeline canonicalizes outputs into an evaluation graph before aligning prediction units to ground truth.Source: §A.2.5 · pp. 27–28

B.1 Intra-Family Metric Correlation

Metric correlations show that OmniCapBench's structural and semantic measures capture distinct capabilities. A diagnostic subset further shows that localized evaluation can distinguish models that global text scores treat as nearly equivalent.

B.1 Intra-Family Metric Correlation

  • Event F1 and Shot F1 correlate at r = 0.25, while CCC and Shot F1 correlate at r = 0.17, separating audio perception and identity tracking from visual temporal grounding.EVSA correlates with its audio and visual prerequisites at r = 0.67 and r = 0.62, respectively.Source: §B.1 · p. 29, Figure B.1 · p. 29
  • Subject Precision and Subject Recall correlate at r = −0.03, indicating that hallucinated descriptions and coverage of ground-truth facts vary independently.Audio descriptions and Scene Recall are also uncorrelated (r = −0.02).Source: §B.1 · p. 29, Figure B.1 · p. 29

B.2 Detailed Setup

  • The 200-video diagnostic subset selected cases where models’ maximum global text-score difference was under 10%, making them nearly equivalent under whole-caption scoring.The models were then reevaluated on those same videos using OmniCapBench’s structural metrics.Source: §B.2 · pp. 29–30
  • Localized metrics revealed underlying quality gaps among models that appeared identical under global text scoring.The results indicate that global text metrics can reward linguistically fluent but structurally flawed outputs.Source: §B.2 · p. 30

B.3 Full Main Results

Expanded submetrics reveal divergent detection strategies, weak fine-grained temporal boundaries, and missed audio-visual links. Case studies illustrate how structured measures expose identity fragmentation across shots.

B.3 Full Main Results

  • The expanded results also report localized LLM-judge scores and structural and semantic performance stratified across three video-duration buckets: under 1 minute, 1–3 minutes, and 3–5 minutes.The captions specify the table scopes but do not provide additional outcome values in these passages.Source: §B.3 · p. 30, Table B.4 · p. 31, Table B.5 · p. 32
  • Gemini 3.1-Pro has Reference Recall of 77.02% and Reference Precision of 70.56%, while Gemini 2.5-Pro has higher Reference Precision at 71.39% but lower recall.Shot tIoU exceeds 70%, but Gemini 3.1-Pro’s Subshot Precision falls from 88.77% for Shots to 60.29% for Subshots.Source: §B.3 · p. 30, Table B.2 · p. 31
  • Gemini 3.1-Pro’s EVSA Precision is 78.35% and EVSA Recall is 42.14%, with low recall indicating that omitted event-shot links are the primary cross-modal failure.The expanded tables report component precision, recall, F1, and temporal-overlap metrics; the Audio-Visual results isolate Event-Shot Association.Source: §B.3 · p. 30, Table B.3 · p. 31

B.4.1 Identity Persistence: Cross-Shot Coreference Consistency

  • Qwen3-Omni-Instruct scores 30.77 CCC, compared with scores above 76 for three other models, indicating greater identity fragmentation across shots.CCC measures whether the same entity retains a stable identity assignment across multiple shots, rather than whether it is detected at least once.Source: §B.4.1 · pp. 31–33, Figure B.2 · p. 33

B.4.2 Audio-Visual Association: Event-Shot Association

Event-Shot Association tests whether recognized audio events link to the visual shots supporting their sources. Case-study predictions reveal distinct omissions and false links that change which visual evidence supports each event.

B.4.2 Audio-Visual Association: Event-Shot Association

  • EVSA checks whether an audio event connects to the visual shot or shots supporting its source, distinguishing audio recognition from grounding that event in the visual timeline.Dialogue spanning almost the full clip should link to all five shots, while the mechanical sound from 5.5 to 8.0 seconds is supported by Shot 4.Source: §B.4.2 · pp. 33–34
  • Small graph edits change which visual evidence supports each audio event, so a holistic caption can miss whether links are omitted or temporally over-extended.The case study contrasts a missed SFX_1 → Shot 4 edge with the unsupported SFX_1 → Shot 3 edge.Source: §B.4.2 · pp. 33–34
  • Gemini 3.1-Pro recovers all six event-shot links, while other models omit localized sound or dialogue links, or add an unsupported neighboring-shot link.Gemini 2.5-Pro and Qwen3.5-Omni-Plus miss the mechanical-sound link; Qwen3.5-Omni-Flash adds a false link and misses later dialogue coverage.Source: §B.4.2 · pp. 33–34

B.5 Analysis of Specialized Captioning Models

Caption-specialized models often fail to produce OmniCapBench’s connected Reference–Event–Shot structure, so direct structural scores can conflate perception with schema compliance. Post-hoc parsing repairs format but not missing grounding, while reported model rankings are largely stable across matching-threshold changes.

B.5 Analysis of Specialized Captioning Models

  • Caption-specialized models exhibit schema rejection, fixed caption templates, incomplete graph construction, or degenerate repeated fragments, rather than usable connected evaluation graphs.ASID-Caption may emit timestamped prose; TimeChat-Captioner uses its own JSON fields; AVoCaDO leaves events empty and links shots to an undefined reference; UGC-VideoCaptioner repeats zero-duration spans.Source: §B.5 · p. 34, Figure B.4 · p. 35
  • These failures arise at instruction following and structural generation, before semantic matching, so downstream structural scores would conflate video perception with schema compliance.The authors therefore exclude highly specialized captioners from primary benchmarking because their supervised fine-tuning biases them toward holistic free-form paragraphs.Source: §B.5 · p. 34

B.6 Post-hoc Parsing Baseline

  • The fixed post-hoc parser converts free-form captions into units and marks unsupported fields unknown, making specialized captioners evaluable rather than scoring them zero.Tables B.8 and B.9 distinguish rule-based grounding metrics from localized semantic scores in the native-versus-post-hoc comparison.Source: §B.6 · p. 36, Table B.8 · p. 36, Table B.9 · p. 37
  • Post-hoc parsing improves structural well-formedness but degrades grounding-dependent metrics, including Gemini 3.1-Pro’s CCC from 34.43 to 5.20 and Qwen3-Omni-Instruct’s Event F1 from 41.70 to 12.82.Gemini’s EVSA falls from 51.46 to 38.29 and its Shot score from 59.05 to 35.84; Shot F1 and Shot tIoU exceed 95 for every post-hoc entry.Source: §B.6 · p. 36, Table B.8 · p. 36
  • ASID-Captioner-7B reaches 96.17 SGC after post-hoc parsing but remains at 9.65 CCC, 11.09 Event F1, and 10.87 EVSA.The video-blind parser contributes no visual evidence, and cross-shot identity and audio-visual relations absent from the original dense captions cannot be reconstructed afterward.Source: §B.6 · p. 36

B.7 Threshold Sensitivity

  • Across threshold sweeps, Spearman’s ρ against the default is 1.000 for every metric except Shot F1 (0.971) and Speaker (0.914), with no systematic open- versus closed-source reversal.Tightening shot tIoU from 0.30 to 0.50 costs Gemini 3.1-Pro 12.2 Shot F1 points; the only top-1 change is a 0.02-point Speaker near tie.Source: §B.7 · p. 37, Table B.10 · p. 37

C Limitations and Broader Impact

OmniCapBench’s conclusions are bounded by its annotation scope, video duration, possible evaluator–model bias, and strict schema. The paper also notes privacy, copyright, and interpretation considerations for a benchmark built from video data.

C Limitations and Broader Impact

  • The benchmark scores structural fidelity against its annotated reference system, so valid peripheral video details outside annotation scope go uncredited.Its dataset is limited to videos under five minutes; substantially longer videos may require identity-tracking architectures beyond the paper’s scope.Source: C Limitations and Broader Impact · p. 37
  • Reference drafts from frontier models, including some evaluated model families, leave evaluator–model bias possible and may slightly favor models sharing a drafter’s stylistic priors.The authors use multiple model families, deterministic validators, and human arbitration, but do not claim the bias is fully eliminated.Source: C Limitations and Broader Impact · pp. 37–38
  • The strict deep-structured JSON schema disadvantages models optimized for free-form prose, and low benchmark scores do not necessarily indicate inability to produce helpful open-ended summaries.The benchmark is intended to measure structured audio-visual understanding and grounded reasoning.Source: C Limitations and Broader Impact · p. 38, D Ethical Considerations · p. 38

D Ethical Considerations

  • The benchmark uses publicly accessible media and academic-dataset videos, filters sensitive content, and requires users to follow source-platform privacy guidelines.It distributes annotations and download scripts rather than raw video files, respecting creators’ distribution rights.Source: D Ethical Considerations · p. 38

E Dataset Licenses and Usage

The appendix distinguishes benchmark and software licensing from source-video terms and documents prompting and scoring instructions for native tracks and localized subshot checks.

E Dataset Licenses and Usage

  • OmniCapBench permits academic and commercial use under MIT, but source-video licenses generally restrict use to non-commercial research and raw MP4 files are not hosted or distributed.A deterministic script downloads the publicly available videos from their original URLs.Source: E Dataset Licenses and Usage · p. 38

F Instruction Templates and Prompts

  • The generation prompt requests JSON with persistent references, timestamped audio events, and non-overlapping shots linked to active events and references.Reference entries include semantic descriptions and appearance anchors; event entries include time ranges and spoken dialogue transcripts when applicable.Source: F Instruction Templates and Prompts · p. 39
  • The post-hoc parser prompt extracts facts from dense captions into the MTSS JSON schema, assigning IDs and linking references, events, and shots.It may infer timestamps when mentioned and omit or estimate exact timestamps when absent.Source: F Instruction Templates and Prompts · p. 39
  • Recall checks whether predictions cover ground-truth facts, while precision checks whether ground truth supports prediction facts; contradictions count against coverage.The prompts score only the specified description fields and exclude other fields such as timestamps, events, and shots.Source: F Instruction Templates and Prompts · p. 41

Subshot Formatting and task_text Inserted by the Code

  • Subshot matching requires a counterpart to preserve the main entities, visible state, action or pose, and local scene facts; quality scoring checks target facts against context.Subshot content may include entity visibility, pose or action, spatial relations, interactions, motion, and transient visual states.Source: Event Quality Instruction Example · pp. 43–44
  • Shot-fact prompts assess camera and shot-transfer fields, including scale, angle, movement, framing, transitions, and continuity, with contradictions penalized more than omissions.The prompt excludes visual descriptions, active events, references, event content, and timing from this check.Source: Event Quality Instruction Example · p. 45
Loading 2610.12458v1…