PaperlayerView exact revision on arXiv

Source-linked AI summary

RVG-Bench: A Benchmark for Reasoning-Driven Audio-Visual Temporal Grounding

Yue Feng, Boshen Xu, Zihao Yue, Xiaokun Liu, Miao Yang, Xin Tao, Pengfei Wan, Qin Jin

arXiv:2610.32268v1cs.CV

TL;DR

Existing temporal video grounding benchmarks largely use descriptive queries, leaving intent-driven localization underexplored. RVG-Bench introduces an audio-visual benchmark whose queries require reasoning to infer targets. Across 22 models, the strongest achieves 49.6 mIoU, and the benchmark currently covers short-to-medium videos rather than long-form content.

  • Problem

    Existing temporal grounding benchmarks primarily use descriptive visual queries, while real-world requests often require inferring intent to locate evidence, causes, or remembered moments.

    Source: §1 · p. 2

  • Method

    RVG-Bench uses indirect queries requiring reasoning before localization, organized across eight reasoning subcategories and incorporating audio evidence in more than half of queries.

    Source: §1 · pp. 2–3

  • Results

    49.6 mIoU is the highest score among 22 evaluated models, achieved by Qwen3.8-Omni-Flash; the strongest locally evaluated baseline reaches 30.1 mIoU.

    Source: §1 · p. 3

  • Takeaways & Limitations

    RVG-Bench exposes limitations in target inference and temporal localization that descriptive benchmarks largely overlook.

    Source: §5 · p. 12

    The benchmark focuses on short-to-medium videos and does not evaluate movie-level narrative understanding or long-term memory.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Existing temporal video grounding (TVG) benchmarks primarily focus on localizing target video segments according to descriptive text queries of the visual content. However, real-world queries are more diverse and often require the model to first understand the user's intent before localization, such as finding visual evidence, identifying the cause of an event, or recalling a remembered moment, yet such queries remain largely overlooked. To mitigate this gap, we introduce RVG-Bench, a benchmark for reasoning-driven audio-visual temporal grounding, where every query is constructed such that the target segment can only be identified through reasoning about the query's underlying intent. We carefully construct the benchmark and rigorously verify its data quality to ensure models truly comprehend the audio-visual content. RVG-Bench comprises 1,255 videos and 1,890 query-interval pairs across eight reasoning types. We evaluate 22 models and find that the strongest, Qwen3.8-Omni-Flash, achieves only 49.6 mIoU. Through a series of controlled analyses, including oracle-grounding probes and test-time scaling experiments, we systematically examine where and why current models struggle, providing insights for future model development.

1 Introduction

Existing temporal grounding benchmarks largely reduce localization to matching visual descriptions, leaving intent-driven queries unevaluated. RVG-Bench makes target inference central, adds audio-visual evidence, and shows that current models still struggle.

1 Introduction

  • Existing benchmarks primarily test descriptive matching, while real-world queries often ask for evidence, causes, or remembered moments rather than describing a segment.For example, locating the action that caused a burglar to fall requires inferring the earlier action—pouring water on the stairs—and then localizing it.Source: §1 · pp. 1–2
  • RVG-Bench requires models to infer the user’s intent before localizing a target that cannot be identified through surface-level semantic matching.Its structured taxonomy covers relational, intrinsic-attribute, and relative-attribute reasoning across eight subcategories; more than half of queries require audio evidence alone or jointly with vision.Source: §1 · pp. 2–3
  • The benchmark contains 1,255 videos and 1,890 query–interval pairs, built through a four-stage pipeline with full human validation.Construction includes strict information-leakage prevention and multi-stage verification to ensure each query has a unique, unambiguous target segment.Source: §1 · p. 3
  • 49.6 mIoU is the strongest result among 22 evaluated models, achieved by Qwen3.8-Omni-Flash; the strongest locally evaluated baseline reaches 30.1 mIoU.The evaluation covers vision-language and omni-modal architectures and includes controlled analyses of target inference, localization, and test-time task decomposition.Source: §1 · p. 3

2 Related Work

Prior temporal grounding benchmarks are predominantly vision-centric and descriptive, although some newer work incorporates audio. Existing approaches that add reasoning-related tasks retain limitations in query format, reasoning coverage, or modality.

2 Related Work

  • Widely used TVG datasets such as Charades-STA, ActivityNet Captions, and QVHighlights are primarily vision-centric, while newer efforts incorporate audio signals [7].Traditional queries describe visually observable actions or events, often in silent videos.Source: §2 · p. 3
  • NEXT-GQA and REXTIME combine question answering with localization, but their multiple-choice format leaks answer cues and departs from open-ended queries.The passage also notes that these benchmarks are vision-centric and overlook audio.Source: §2 · p. 4
  • ToTG-Bench [20] explores task-oriented localization but lacks systematic emphasis on the complex reasoning needed to resolve queries.Existing TVG benchmarks more broadly remain predominantly descriptive, leaving reasoning-driven localization largely unexplored.Source: §2 · p. 4

3 RVG-Bench

RVG-Bench defines grounding queries that require reasoning from indirect intent to a target, then builds and validates a benchmark spanning varied reasoning types and audio-visual evidence. Its videos are long relative to the target intervals, and many queries require audio evidence.

3.1 Benchmark Design

  • RVG-Bench organizes indirect target inference into relational, intrinsic-attribute, and relative-attribute reasoning, with multi-step queries chaining two or more atomic operations.Relational queries traverse causal, temporal, or referential dependencies; intrinsic-attribute queries infer implicit properties; relative-attribute queries compare candidate segments.Source: §3.1 · p. 4
  • The reasoning taxonomy covers relational, intrinsic-attribute, and relative-attribute queries, with eight fine-grained subcategories and audio included as a source of evidence.Multi-step queries compose two or more atomic operations to identify a target.Source: §3.1 · pp. 4–5

3.2 Benchmark Construction

  • The construction pipeline combines caption-based query proposal, MLLM quality refinement and interval adjustment, difficulty-aware sampling, and full human validation.Proposal uses visual, audio, and joint audio-visual captions; human reviewers remove invalid samples, rewrite some questions, and refine temporal boundaries.Source: §3.2 · pp. 5–7, Figure 2 · p. 6
  • Candidate queries are generated from timestamped dense captions, and an event graph supports construction of multi-step queries by composing event nodes and relations.The proposal stage produces approximately 6.5k single-step and 1.1k multi-step candidates, each paired with a draft interval.Source: §3.2 · p. 6
  • Quality refinement checks answerability, uniqueness, logical soundness, and information leakage before sampling and human review.After automated refinement, 66.3% of proposed samples are retained; humans review all 2,500 sampled candidates against the same four criteria.Source: §3.2 · p. 7

3.3 Statistics

  • 1,890 query-interval pairs across 1,255 untrimmed videos comprise the benchmark, including 1,588 single-step and 302 multi-step queries.The construction process samples 2,500 candidates for human review; 610 are removed, leaving the final set.Source: §3.2 · p. 7, §3.3 · p. 7
  • 108.39 seconds is the average video duration, while target intervals average 9.46 seconds and occupy about 9.6% of video duration.Ground-truth segments are approximately uniformly distributed across the timeline, reducing the usefulness of simple beginning-or-end positional priors.Source: §3.3 · pp. 7–8
  • 45.34% of queries are visual-only, 11.38% audio-only, and 43.28% require joint audio-visual reasoning.Thus, more than half require audio evidence, alone or together with vision.Source: §3.3 · p. 8

4 Experiment

Experiments evaluate 22 multimodal models on RVG-Bench and find substantial difficulty across benchmark analyses and model diagnostics. The leading model remains below the human reference, oracle target descriptions help, and test-time reasoning strategies do not consistently improve results.

4.1 Setup

  • 22 models are evaluated using mIoU and IoU@0.3, with local models run using official code and checkpoints and API models receiving visual or audio-visual inputs.Chain-of-Thought prompting is used unless otherwise specified.Source: §4.1 · p. 8

4.2 Benchmark Analysis

  • 44.4 versus 30.1 mIoU separates Gemini-3-Pro and TimeLens-8B on RVG-Bench, wider than their 2.5-point gap on three descriptive benchmarks.Removing audio reduces Gemini-3-Pro from 44.4 to 38.4 mIoU on RVG-Bench, with larger drops on audio-required and joint-evidence subsets.Source: §4.2 · pp. 8–9, Table 2 · p. 9, Table 3 · p. 10
  • 65.3 mIoU and 78.5 IoU@0.3 are the human averages on a stratified 200-example audit, with scores of 68.5 on single-step and 52.4 on multi-step queries.The human baseline averages two annotators’ IoUs with ground truth for each example before aggregation.Source: Table 2 · p. 9, §4.2 · p. 10

4.3 Model Analysis

  • 49.6 mIoU is Qwen3.8-Omni-Flash’s leading score, below the human reference of 65.3 mIoU; TimeLens-8B leads locally at 30.1 mIoU.The leading model reaches 68.0 IoU@0.3, while the human reference is 78.5 IoU@0.3 on the 200-example audit subset.Source: Table 2 · p. 9, §4.3 · p. 10
  • Oracle target descriptions raise mean mIoU from 26.9 to 36.8 across 15 analyzed models, while the 22-model comparison rises from 21.2 to 29.6.Even with the target supplied, Qwen2.5-Omni and MiniCPM-O-2.6 reach only 9.0 and 4.3 mIoU, respectively.Source: §4.3 · p. 11, Figure 3 · p. 11
  • Causal queries are difficult, and multi-step scores fall from 46.1 to 35.8 mIoU for Gemini-3-Pro and from 31.9 to 21.3 for TimeLens-8B.Across most models, Seq and Comp are generally stronger categories than causal queries.Source: §4.3 · p. 11
  • No test-time strategy consistently dominates: 2-step is best for Gemini-3-Pro, CoT for Qwen3.8-Omni-Flash, and Direct for Qwen3.8-Flash.For Qwen3.8-Flash, Direct leads on single-step queries while 2-step leads on multi-step queries.Source: Figure 3 · p. 11, §4.3 · p. 12

5 Conclusion

RVG-Bench shifts temporal grounding from descriptive matching to reasoning-driven localization over joint audio-visual evidence. Its analyses expose challenges in target inference and temporal localization that existing benchmarks largely overlook.

5 Conclusion

  • RVG-Bench requires inferring implicit targets through diverse reasoning types over joint audio-visual evidence, exposing limitations in current models.The analyses include category breakdowns, oracle-grounding probes, and test-time scaling.Source: §5 · p. 12

Limitations

The benchmark focuses on short-to-medium videos; long-form sources and their associated narrative and memory demands remain outside its evaluation scope.

Limitations

  • RVG-Bench covers short-to-medium videos, not full-length movies or hourlong uncut footage, so it does not evaluate movie-level narrative understanding or long-term memory.The authors identify extending reasoning-driven grounding to long-form content as future work.Source: Limitations · p. 12

A.1 Comparison with Representative Benchmarks

The benchmark comparison reports test-split statistics and whether representative datasets cover audio, reasoning attributes, and multi-step queries; the taxonomy table pairs each subcategory’s definition with an example query.

A.1 Comparison with Representative Benchmarks

  • Table 4 compares representative TVG benchmarks using test-split statistics and indicators for audio, relational reasoning, attributes, and multi-step queries.A dash marks statistics unavailable from the original paper or because the dataset is not open source.Source: Table 4 · p. 16

A.2 Full Reasoning Taxonomy with Examples

  • Table 5 organizes RVG-Bench’s reasoning taxonomy by subcategory, with one row defining each indirect target specification and another giving an example query.Subcategory abbreviations follow Table 1.Source: Table 5 · p. 17

A.3 Benchmark Statistics and Distributions

The appendix checks temporal-position bias, describes full human review and timestamp refinement, reports a human grounding audit, and illustrates the benchmark’s reasoning categories through examples.

A.3 Benchmark Statistics and Distributions

  • The ground-truth interval centers, starts, and ends broadly cover the timeline, reducing the usefulness of simple temporal-position priors.Figure 4 plots the normalized distributions of all three interval positions relative to video duration.Source: §A.3 · pp. 17–18, Figure 4 · p. 18

A.4 Full Human Validation

  • Of 2,500 candidates reviewed by one graduate-student annotator, 610 (24.40%) were removed and 1,890 retained; among retained samples, timestamps were refined for 240 (12.70%) and questions rewritten for 98 (5.19%).Source: §A.4 · pp. 18–19

A.5 Human Baseline

  • 65.3 mIoU and 78.5 IoU@0.3 are the human baseline scores on a 200-example audit subset; human–human agreement is 73.7 mIoU.Two annotators independently marked timestamps for each example, and the baseline averages their IoUs with ground truth.Source: §A.5 · p. 20, Table 7 · p. 21

A.6 Additional Benchmark Cases

  • The examples span relational dependencies, implicit character or scene attributes, and relative ranking or order, while multi-step cases combine inferences such as causal reasoning and anomaly detection.The multi-step example locates the protagonist’s discovery of an empty closet after inferring a duplication mechanism and its failure.Source: §A.6 · p. 21, Figure 6 · p. 22, Figure 7 · p. 23

B Evaluation Protocol

Evaluation uses model-specific official or default input protocols and common temporal-grounding metrics; the setup compares complete systems rather than isolating architecture or reasoning under equal frame or token budgets.

B.1 Model Input and Sampling Settings

  • The comparison characterizes complete systems under their supported settings, not architecture or reasoning ability at an equal frame or token budget.Models use model-specific official or default input protocols and common temporal-grounding metrics.Source: §B.1 · p. 25
  • API-based evaluations use model-specific video sampling and timestamp protocols, with temperature 0 and a 1,024-token output limit for the described settings.Gemini-3-Pro samples at 1 FPS; GPT-5.5 uses timestamped frames with duration-dependent sampling and frame grouping.Source: §B.1 · p. 25
  • The evaluation varies modality and prompting for selected API models, while open-source omni and vision-language models use their specified frame-sampling and token or resolution settings.Selected API models receive direct, CoT, 2-step, and oracle-grounding prompts; invalid timestamps and failed requests score zero.Source: §B.1 · p. 25

C.1 Comparison with Descriptive Grounding Benchmarks

The appendix includes a comparison of TimeLens-optimized benchmarks with RVG-Bench and a modality ablation across Charades-TimeLens and RVG-Bench.

C.1 Comparison with Descriptive Grounding Benchmarks

  • Table 8 compares reported results on TimeLens-optimized benchmarks with RVG-Bench, including the mIoU gap between Gemini and TimeLens-8B.The caption distinguishes Gemini-2.5-Pro on the first three benchmarks from Gemini-3-Pro on RVG-Bench and defines relative gap as a percentage of the TimeLens-8B score.Source: Table 8 · p. 26

C.2 Modality Ablation

  • Table 9 reports mIoU for audio-plus-visual, visual-only, and audio-only inputs on Charades-TimeLens and RVG-Bench, with RVG-Bench also divided by modality requirement.The RVG-Bench breakdown covers visual-required, audio-required, and joint audio-visual-required subsets, as well as the overall score.Source: Table 9 · p. 27

C.3 Full Oracle-Grounding Results

Oracle target descriptions generally improve grounding, but models with weak localization still struggle even when the target is supplied.

C.3 Full Oracle-Grounding Results

  • The 22-model mean rises from 21.2 to 29.6 mIoU with oracle target descriptions.Gemini-3-Pro improves from 44.4 to 63.9 mIoU, and TimeLens-8B from 30.1 to 45.4.Source: §C.3 · p. 27, Table 10 · p. 28
  • Qwen2.5-Omni, HumanOmniV2, and MiniCPM-O-2.6 reach only 9.0, 6.4, and 4.3 mIoU, respectively, even with the correct target supplied.The results indicate that explicit target content helps, while accurate temporal localization remains necessary.Source: §C.3 · p. 27

C.4 Full Task-Decomposition Results

Task decomposition modestly favors multi-step queries, stricter overlap thresholds expose boundary errors, and self-verification benefits vary across models.

C.4 Full Task-Decomposition Results

  • 2-step prompting improves multi-step mIoU over CoT for 10 of 15 models, with a +0.83-point mean change versus −0.05 on single-step queries.The overall gain is 0.09 points; relative to Direct, 2-step improves multi-step mIoU for 10 of 15 models by 1.22 points on average.Source: §C.4 · p. 28, Table 11 · p. 29
  • Gemini-3-Pro reaches 49.8 mIoU with 2-step prompting, versus 43.2 with Direct and 44.4 with CoT.Qwen3-Omni and TimeSuite largely recover performance lost under CoT, while 2-step scores remain below CoT for four models despite oracle-description gains of 9.5, 11.6, 10.6, and 14.7 points.Source: §C.4 · pp. 28–29, Table 11 · p. 29

C.5 Recall at Stricter IoU Thresholds

  • Qwen3.8-Omni-Flash recall falls from 68.0% at IoU ≥0.3 to 51.8% at IoU ≥0.5 and 33.4% at IoU ≥0.7.Stronger coarse localization does not guarantee precise event boundaries.Source: §C.5 · p. 29, Table 12 · p. 30

C.6 Self-Verification

  • Gemini-3-Pro improves from 44.4 mIoU at N = 1 to 50.0 at N = 10, while TimeLens-8B and ChronusOmni show no consistent upward trend.N is the number of candidate predictions; N = 1 matches the CoT result in Tab. 2.Source: §C.6 · p. 31, Figure 10 · p. 32
  • More candidate sampling alone is insufficient when a model cannot reliably select a correct reasoning path and aligned interval.The results suggest self-verification is promising for sufficiently capable proprietary models, while open-source systems need improved verification or self-assessment.Source: §C.6 · p. 31

C.7 Error Diagnosis

This section diagnoses perception, temporal-localization, and reasoning failures, including recurring localization difficulty and successful intervals paired with flawed reasoning. It also describes the prompts and construction pipeline used to create and verify reasoning-dependent queries, their quality, temporal boundaries, and modality requirements.

C.7 Error Diagnosis

  • Diagnosis uses a stratified subset of approximately 700 examples, four error categories, and IoU ≥0.5 for successful localization; Gemini-3-Pro judges default-CoT failures across three settings.For predictions with valid diagnostic evidence in all settings, the analysis assigns the earliest failed stage in the order understanding, perception, reasoning, localization.Source: §C.7 · p. 32
  • Perception errors account for 44.9% of ChronusOmni’s failures, 45.9% of Qwen3-Omni’s, 53.0% of Qwen2.5-Omni’s, and 79.9% of Video-R1-7B’s.The findings point to difficulty extracting subtle, temporally dispersed evidence, including for systems with both audio and video access.Source: §C.7 · pp. 32–33
  • Localization is the most frequent GPT-5.5 error label for Qwen3.8-Omni-Flash, Qwen3.8-Flash, and Seed2.0-Lite, at 34.6%, 33.2%, and 38.6% of valid judgments.Seed2.0-Pro has comparable localization and reasoning frequencies, 29.0% and 28.9%; the findings suggest target inference alone is insufficient without temporal alignment.Source: §C.7 · p. 33, Table 13 · p. 33
  • Primary-label agreement is 80.11% for Gemini-3-Pro and 70.03% for Qwen3-VL-8B, with Cohen’s κ of 0.704 and 0.600, respectively.Both judges rank primary errors localization > perception > reasoning > understanding for both models, though localization and perception are close for Qwen3-VL-8B, particularly under GPT-5.5.Source: §C.7 · p. 33, Table 14 · p. 34
  • Among successful localizations, Qwen2.5-Omni and TimeSuite receive wrong-reasoning labels for 25.0% and 27.5% of predictions, respectively.The labels distinguish correct intervals supported by flawed intermediate reasoning from successes supported by a correct reasoning path.Source: §C.7 · p. 33

D.1 Evaluation Prompts

  • Evaluation prompts cover Default Chain-of-Thought, Direct, Two-step, and Ground (Oracle Answer), with adaptations for model capabilities and official formats.Vision-language models are restricted to visual cues, Gemini models use MM:SS timestamps, and specified reasoning models wrap reasoning and predictions in <think> and <answer> tags.Source: §D.1 · pp. 35, 37

D.2 Construction Prompts

  • Construction prompts first generate continuous timestamped visual, audio, and audiovisual captions, then propose single-step query-answer pairs across reasoning subcategories.Queries must require inference rather than direct matching, avoid directly describing the target, resolve to one segment, and be verifiable from context.Source: Dataset Construction Prompts · pp. 37–39
  • For multi-step queries, construction introduces an event-graph representation to model inter-event dependencies before query generation.The graph fuses multi-view captions with plot summaries and captures event functions, internal states, and relations such as causal links, setup-payoff, contrast, and similarity.Source: Dataset Construction Prompts · pp. 39–40
  • Questions combine reasoning types so each phrase is necessary and their intersection uniquely identifies the target segment.The construction guidance requires at least two reasoning steps and tests each phrase by removing it to detect ambiguity or a shifted target.Source: Dataset Construction Prompts · p. 40
  • Candidate queries are checked for answerability, uniqueness, logical soundness, and absence of information leakage.The answer must describe a concrete, temporally localizable event, and the question must not directly describe the target’s visual or audio content.Source: Dataset Construction Prompts · pp. 41–42
  • Temporal refinement prompts seek the minimal continuous interval necessary and sufficient for the answer, with boundaries tied to the beginning and end of answer-defining evidence.Annotators may shrink the coarse window or shift slightly outside it, while avoiding unnecessary lead-in or aftermath.Source: Dataset Construction Prompts · p. 42
  • A modality-verification prompt classifies each QA pair as requiring visual information, audio information, or both to answer and identify the segment.It asks for a brief explanation of the modality requirement for each pair.Source: Dataset Construction Prompts · p. 43

Related papers

Loading 2610.32268v1…