PaperlayerView exact revision on arXiv

Source-linked AI summary

DVBench: Benchmarking MLLMs for Understanding Dynamic Charts and Narratives in Data Videos

Bomiao Wang, Zekai Shao, Jiexiang Lan, Xiaoliang Fu, Xingchen Zeng, Siming Chen

arXiv:2608.29711v1cs.CL

TL;DR

Existing evaluations largely separate chart comprehension from video understanding, leaving dynamic data storytelling insufficiently assessed. DVBench benchmarks this setting across five dimensions and 1,000 verified questions, finding Gemini-3.1-Pro performs best overall while Kimi-k2.5 leads open-source models.

  • Problem

    Existing benchmarks largely isolate static chart comprehension and natural-scene video understanding, leaving comprehension of dynamic data-driven storytelling insufficiently evaluated.

  • Method

    DVBench evaluates data video understanding across five dimensions using 300 videos and 1,000 expert-verified QA pairs curated through a semi-automatic pipeline.

  • Results

    Gemini-3.1-Pro achieves the best overall performance, while Kimi-k2.5 is the leading open-source model; performance does not scale strictly with size or narrative proficiency.

  • Takeaways & Limitations

    Fine-grained analyses and ablations expose dimension-specific weaknesses and show that frame density and subtitle inputs affect MLLM performance.

  • Takeaways & Limitations

    Reliance on human review ensures data fidelity but currently limits DVBench’s scale.

Abstract

from arXiv · show

While MLLMs have made significant strides in chart comprehension and video understanding, current evaluations largely isolate these capabilities, leaving a critical gap in understanding temporally evolving structured visual information. To address this gap, we introduce DVBench, a benchmark for evaluating MLLMs on data videos, a storytelling medium that integrates dynamic charts with structured narratives. We decompose data video understanding into five dimensions. DVBench comprises 300 real-world data videos and 1,000 human-verified QA pairs curated through a rigorous semi-automated pipeline. Extensive evaluations of nine MLLMs show that Gemini-3.1-Pro achieves the best overall performance, while Kimi-k2.5 is the strongest open-source model. We further identify two notable phenomena: open-source model performance does not scale strictly with parameter size, and narrative proficiency does not guarantee visual capability. Fine-grained analyses and ablation studies further reveal dimension-specific weaknesses and the effects of frame configurations and subtitle inputs, informing future MLLM development. DVBench is publicly available at https://bomiaowang.github.io/DVBench/.

1 Introduction

DVBench addresses the gap between static chart and natural-scene video benchmarks by evaluating MLLMs on data videos, which combine dynamic visualizations with narrative structures and cinematic transitions. It introduces a five-dimensional benchmark with 300 data videos, 1,000 human-verified QA pairs, and extensive model analyses.

  • Motivation: DVBench fills a benchmark gap because chart evaluations are static, whereas video benchmarks largely target physical entities and actions rather than dynamic structured information over time.Data videos bridge these settings by integrating dynamic visualizations, narrative structures, and cinematic transitions.
  • Benchmark Design: DVBench evaluates data-video understanding across Narrative, Animation, Chart Perception, Chart Reasoning, and Alignment.These dimensions cover video themes, dynamic transitions, visual identification, data-insight extraction, and cross-modal alignment.
  • Dataset: The benchmark contains 300 high-quality data videos and 1,000 QA pairs created through a rigorous semi-automatic pipeline with human verification.This dataset supports systematic evaluation of the proposed data-video-understanding task.
  • Experimental Findings: Gemini-3.1-Pro achieves the best overall performance among nine evaluated MLLMs, while Kimi-k2.5 is the leading open-source model.The study also finds that data-video understanding does not scale strictly with model size, and narrative proficiency does not guarantee fine-grained visual capability.
  • Fine-Grained Analysis and Ablations: Fine-grained analyses identify weaknesses in non-linear animation pacing and dynamic-chart counting, while denser frame sampling is not always beneficial and subtitles improve performance.Robustness to increasing video length varies across models.

2 Related Works

Existing benchmarks separately evaluate chart and video understanding, while data-video research studies the integration of dynamic visualizations with structured narratives. DVBench addresses this gap by providing a framework for understanding data-driven narratives.

  • Chart understanding: Chart-understanding benchmarks span structural extraction, reasoning, captioning, and increasingly sophisticated expert-validated questions across real-world topics.Early chart QA focused on scientific data in open-vocabulary settings, while later work expanded reasoning logic and topic diversity.
  • Video understanding: Video-understanding benchmarks have progressed from short clips and static perception toward longform videos, dynamic state changes, causal reasoning, temporal logic, and specialized narratives.Domain-specific benchmarks cover cinematic plot progression as well as multidisciplinary knowledge and scientific processes.
  • Data video studies: DVBench establishes a framework for evaluating understanding of data-driven narratives, complementing existing multimodal benchmarks with this integrated setting.Table 1 compares DVBench with existing multimodal understanding benchmarks and indicates whether they assess multi-dimensional capabilities.
  • Data video studies: Data videos combine dynamic visualizations with structured narratives to engage audiences and convey rich information across diverse domains.Prior research examines visual-animation interplay and scene-semantic alignment as foundational design elements.

3 DVBench

DVBench decomposes data-video understanding into five dimensions spanning narrative, animation, chart reading and reasoning, and chart–narration alignment. It contains 300 diverse data videos and 1,000 curated QA pairs generated through dimension-specific procedures and expert review.

  • 3.1 Evaluation Dimensions: DVBench evaluates Narrative, Animation, Chart Perception, Chart Reasoning, and Alignment as five complementary dimensions of data-video understanding.Chart Reasoning distinguishes inferred insights from directly readable graphical information, while Alignment uses subtitle cloze tasks to test chart–narration correspondence.
  • 3.2 Benchmark Construction: The benchmark comprises 300 data videos spanning diverse topics and presentation styles, all featuring dynamic charts and structured narratives.Video lengths range from under 30 seconds to approximately 37 minutes, covering short, medium, and long duration categories.
  • 3.2 Benchmark Construction: QA candidates are generated with dimension-specific strategies, combining MLLM-generated narrative questions, manually designed visual questions, multimodal insight extraction, and subtitle cloze construction.Chart Reasoning includes temporal tasks formed by clustering visual and textual insights by subjects and measures.
  • 3.2 Benchmark Construction: Expert curation establishes a shared annotation protocol through joint viewing and discussion before one annotator constructs the QA set and another independently inspects 10% of pairs.The process is designed to ensure annotation quality after selecting questions from an initial candidate pool.
  • 3.3 Benchmark Statistics: The 1,000 QA pairs are distributed across five dimensions, with the largest share assigned to Chart Reasoning because models remain vulnerable to reasoning despite strong basic perception.Visual questions span common bar and line charts as well as a diverse long tail of chart types.

4 Experiment

DVBench evaluates nine MLLMs across five dimensions using default frame sampling and video-only inputs, with additional frame and subtitle ablations. Proprietary models generally lead, but Kimi-k2.5 is a strong exception, while analyses expose weaknesses in temporal pacing, dynamic counting, grounding, and frame efficiency.

  • Main results: Qwen3.5-27B reaches 62.30% average accuracy and exceeds larger Qwen3.5 mixture-of-experts models, showing that open-source performance does not scale directly with parameter count.Qwen3.5-27B also achieves 74.29% in Narrative but only 59.40% in Animation and 56.83% in Chart Reasoning, revealing stronger macro-narrative than fine-grained visual understanding.
  • Main results: Kimi-k2.5 achieves 75.87% average accuracy, ranking second overall and surpassing Claude-4.6-Sonnet and GPT-5.4, while proprietary models generally perform best.The same pattern holds in Alignment, where proprietary models and Kimi-k2.5 generally outperform other open-source models across text-based, visual-grounding, and human-evaluation metrics.
  • Fine-grained analysis: Animation Timeline questions are especially difficult, with nearly all models below 37.50%, while Aggregation is among the hardest Chart Reasoning categories.Timeline requires perceiving changing animation pace, whereas Aggregation requires counting condition-matching elements across visual states and sometimes multiple frames.
  • Fine-grained analysis: Kimi-k2.5, Gemini-3.1-Pro, and Claude-4.6-Sonnet perform strongest on Alignment, but proprietary models and Kimi-k2.5 score better on Data Insight than Data Context.For other open-source models, the larger Data Insight drop is mainly associated with visual-text grounding errors rather than phrasing.
  • Video length: Leading proprietary models and Kimi-k2.5 remain stable beyond six-minute videos, whereas weaker models decline nearly linearly as video length increases.The effect of video length varies by task, according to the supplied analysis.
  • Ablation studies: Under 64-frame evaluation, model rankings remain consistent, but denser sampling is not monotonically beneficial: 2 fps underperforms 0.5 and 1 fps for all Qwen3.5 models.Subtitles improve every model’s average performance, especially on Narrative and Chart Reasoning, with Qwen3.5 and Gemma gains of 7.92–11.47% in Chart Reasoning.

5 Conclusion

DVBench introduces the first comprehensive benchmark for MLLM understanding of data videos, spanning five dimensions and a rigorously curated dataset of 300 videos with 1,000 expert-verified QA pairs. Evaluations of nine MLLMs show proprietary-model superiority, unexpected parameter-scaling and cross-dimensional patterns, and dimension-specific weaknesses revealed by analyses and ablations.

  • DVBench is the first comprehensive benchmark evaluating MLLMs on data video understanding across five dimensions.
  • The benchmark contains 300 videos and 1,000 QA pairs curated through a semi-automatic pipeline with expert verification.
  • Evaluation of nine MLLMs demonstrates proprietary-model superiority while exposing unexpected parameter-scaling and cross-dimensional proficiency results.
  • Fine-grained analyses reveal weaknesses across evaluation dimensions, and ablations further examine these dimension-specific effects.

Limitations · Appendix · A Potential Risks

DVBench’s reliance on human review limits scalability, while frontier models are relatively saturated on chart perception, motivating harder and broader future benchmark tasks. Although risks are constrained by academic sourcing and quality control, the dataset may inherit bias and expose serious misinformation risks from current models’ hallucinations and weak visual grounding.

  • Limitations: Human review improves data fidelity but currently limits the scale of DVBench.The benchmark uses a semi-automatic construction pipeline combined with expert validation.
  • Limitations: Chart Perception performance is relatively saturated among frontier models, motivating more challenging fine-grained perception tasks.Future iterations should better distinguish model capabilities through harder perception evaluations.
  • Limitations: Future benchmark expansions could cover multilingual data videos, 3D visualizations, and AR infographics.Advancing end-to-end video models may also strengthen fine-grained temporal understanding through denser audio-visual processing.
  • A Potential Risks: Data videos can contain sensitive real-world statistics and analytical narratives reflecting particular socio-economic, geopolitical, environmental, or public-health perspectives.DVBench is intended strictly for academic evaluation, but such content may still create downstream risks during model development and evaluation.
  • A Potential Risks: DVBench’s risks are constrained by sourcing videos from published academic works and manually removing problematic or misleading content where possible.These measures reduce, but do not eliminate, dataset-related risks.
  • A Potential Risks: Prior works may transmit implicit assumptions, framing choices, and subjective interpretations, allowing models to reproduce or amplify inherited biases.This is the dataset’s principal bias-inheritance risk despite sourcing from established academic materials.
  • A Potential Risks: Current open-source MLLMs frequently produce numerical hallucinations and weak visual grounding, risking fabricated or factually incorrect narratives in deployed applications.Potentially affected uses include automated journalism, educational analytics, and data storytelling, where persuasive visualizations could conceal errors.

B Dataset Construction Details … B.3 Annotation Details

DVBench combines licensed academic-use data videos with a semi-automatic construction pipeline that extracts visual, subtitle-based, formalized, narrative, and cross-frame information. Two expert annotators calibrate, curate, and verify questions using explicit evidence-grounding and consistency criteria.

  • B.1 Licenses and Intended Use: The benchmark contains 300 publicly sourced data videos, while copyrights remain with original creators and users must follow applicable laws and hosting-platform terms.DVBench provides a removal mechanism for reasonable third-party copyright requests.
  • B.2 Dataset Construction Prompt: Dataset construction prompts generate narrative-structure questions about visualization intent, visual-narrative mapping, chart transitions, video themes, and subjective attitude.The prompt separates VIS-to-topic, topic-to-VIS, logical, title, and attitude question types.
  • B.2 Dataset Construction Prompt: Visual extraction identifies chart clips and insights such as values, trends, and ranks, requiring answers grounded solely in video content and organized by distinct chart groups.The prompts also require concise, objective answers with deterministic interpretations.
  • B.2 Dataset Construction Prompt: Subtitle extraction captures exact numerical information and qualitative trends, comparisons, intensity, and structural changes, assigning each insight a timestamp and one of ten fact categories.The categories include Value, Proportion, Difference, Distribution, Trend, Rank, Aggregation, Association, Extreme, and Categorization.
  • B.2 Dataset Construction Prompt: Insight formalization converts natural-language facts into structured records containing descriptions, types, parameters, measures, scoped subjects, and timestamps.Subjects encode context filters, breakdown dimensions, and focus groups, while measures preserve complete subtitle wording where possible.
  • B.2 Dataset Construction Prompt: Cross-frame QA generation links facts from different timestamps through shared measures or subjects, requiring traceable evidence and answers expressed as values, deltas, or statuses.The two strategies support cross-attribute calculations and point-to-surface queries anchored by an event or context.
  • B.3 Annotation Details: Two expert annotators calibrated selection criteria on 10 videos, reaching 94.32% raw agreement and Cohen’s κ of 0.88 before resolving disagreements.Questions had to target core insights, use visual or narrative evidence, and require sufficient understanding of the video.
  • B.3 Annotation Details: One annotator constructed the QA set, while the other independently checked 10% for validity, correctness, evidence grounding, and evaluation-dimension consistency through a unified web interface.The interface synchronizes video, subtitles, extracted insights, data clips, QA candidates, and cross-frame reasoning with interactive temporal navigation.

C More Dataset Statistics … D.3 Computational Resources

The appendix reports dataset length statistics, annotation-interface details, model configurations and licensing conditions, and the computational resources used for DVBench evaluation. These details support reproducibility through deterministic decoding, specified visual-input settings, documented access conditions, and a local deployment budget of approximately 240 GPU hours.

  • C More Dataset Statistics: Question and option lengths are summarized across four core evaluation dimensions, while subtitle tables report full and missing subtitle lengths for cloze tasks.Tables 5 and 6 provide these dataset statistics, with Table 5 using mean and min–max range values.
  • D.1 Model Configurations: Evaluation settings specify model versions, visual-input configurations, decoding hyperparameters, and inference backends to support reproducibility.The configuration summary is provided in Table 7.
  • D.1 Model Configurations: Generation is made deterministic by setting temperature to 0, using top-p 1 where applicable, and disabling sampling for locally deployed models through vLLM.The vLLM setting explicitly enforces do_sample=False for open-source models.
  • D Evaluation Details: The annotation interface synchronizes video playback, cross-frame reasoning, subtitles, extracted insights, data clips, and chart-perception QA pairs for efficient verification.An interactive click-to-seek function links insights or questions to corresponding video content.
  • D.1 Model Configurations: Default visual sampling uses 1 fps for Gemini and Gemma, 2 fps for Qwen, and fixed video-frame counts for API-constrained GPT-5.4 and Claude-4.6-Sonnet.GPT-5.4 uses 64 frames and Claude-4.6-Sonnet uses 100 frames per video because of strict API payload limits.
  • D.2 Model Licenses: Model access conditions distinguish proprietary models and Kimi-k2.5, accessed through official APIs, from locally deployed open-source models evaluated under their released licenses.Table 8 summarizes the licenses or access conditions for all evaluated models.
  • D.3 Computational Resources: Open-source models ran locally in BF16 precision on 8×NVIDIA H100 80GB GPUs, requiring approximately 240 GPU hours for evaluation.All inference was conducted strictly for academic research and evaluation.

D.4 Evaluation Metrics · D.5 Evaluation Prompt

DVBench evaluates deterministic, choice-based, and subtitle-completion tasks with exact-match, accuracy, text, visual-grounding, and human-judgment metrics. Standardized prompts constrain answer formatting for each task type, including subtitle-context completion.

  • D.4 Evaluation Metrics: Exact-match tasks normalize model and reference answers before string matching, supporting precise evaluation despite minor formatting differences.Normalization includes lowercasing, punctuation stripping, and article removal.
  • D.4 Evaluation Metrics: Choice-based tasks use accuracy, counting multiple-choice predictions correct only when they exactly match the complete ground-truth option set.This exact-subset criterion disallows both missing and incorrectly selected options.
  • D.4 Evaluation Metrics: Subtitle-completion outputs are assessed through text similarity, visual-text grounding, and human judgment.Text metrics include BLEU-2, METEOR, and BERTScore F1, with BERTScore implemented using roberta-large embeddings and the official bert_score package.
  • D.4 Evaluation Metrics: Visual grounding is measured with EMScore and EMScoreref, combining video-text correspondence with fine-grained frame-word alignment and reference semantics.EMScore evaluates global similarity and frame-word alignment, whereas EMScoreref additionally considers the reference text.
  • D.4 Evaluation Metrics: Human evaluators rate subtitle responses on a five-point scale covering correctness, visual consistency, and faithfulness to the intended message.A score of 1 denotes an incorrect or unsupported response, while 5 denotes a fully correct, visually grounded, faithful response.
  • D.5 Evaluation Prompt: Exact-match prompts request only the answer inside answer tags, excluding units, symbols, and explanations.Examples convert outputs such as 15% or $1 into the bare values 15 or 1.
  • D.5 Evaluation Prompt: Single- and multiple-answer prompts require answer labels inside answer tags, with multiple-answer responses listing all correct labels separated by semicolons.The multiple-answer template explicitly states that more than one option may be correct.
  • D.5 Evaluation Prompt: Alignment prompts provide surrounding subtitle context and ask models to fill the missing narration directly within answer tags.The task combines video content with subtitle context to identify the inserted segment.

E Timeline Question Analysis · F Qualitative Case Studies

The Timeline analysis tests whether sparse frame sampling explains low performance, finding it contributes but is not the sole cause. Qualitative case studies then illustrate model capabilities and failure modes across evaluation dimensions.

  • E Timeline Question Analysis: The study investigates whether Timeline difficulty is primarily caused by sparse frame sampling.
  • E Timeline Question Analysis: The analysis evaluates Timeline questions under different frame conditions available to each model.
  • E Timeline Question Analysis: Frame sampling affects individual Timeline performance, but increasing the frame budget or sampling rate does not consistently resolve the difficulty.Table 9 reports Timeline accuracy under supported sampling rates and each model’s original inference setting.
  • E Timeline Question Analysis: Table 9 compares Timeline performance across different frame sampling conditions and includes each model’s original inference setting.The original setting’s sampling rate or frame budget is shown in parentheses.
  • E Timeline Question Analysis: Sparse sampling is therefore one possible contributor to low Timeline performance rather than its sole cause.
  • F Qualitative Case Studies: Qualitative case studies provide an intuitive view of model capabilities and failure modes across different evaluation dimensions.

F.1 Animation Evaluation Cases · F.2 Chart Reasoning Evaluation Cases

Qualitative cases show that MLLMs recognize explicit visual transformations but struggle with temporal pacing, especially non-linear animation changes. Chart Reasoning examples focus on Aggregation questions.

  • F.1 Animation Evaluation Cases: MLLMs consistently struggle with the Timeline layer of animation.This weakness appears across qualitative examples spanning four editorial layers of animation.
  • F.1 Animation Evaluation Cases: MLLMs can generally identify explicit visual transformations such as color changes or added elements.
  • F.1 Animation Evaluation Cases: Identifying when a chart speeds up is exceptionally challenging.
  • F.1 Animation Evaluation Cases: Identifying when a chart slows down or pauses is exceptionally challenging.
  • F.2 Chart Reasoning Evaluation Cases: Figure 9 illustrates qualitative examples from Aggregation questions in Chart Reasoning.
  • F.2 Chart Reasoning Evaluation Cases: The Chart Reasoning evaluation cases specifically examine Aggregation questions.

F.3 Alignment Analysis · G AI Assistants Usage Declaration

The alignment analysis explains low exact-match scores among open-source models through grounding, compression, paraphrase, and additive-detail errors, while the authors report AI assistance with rigorous manual verification and revision. Animation-timeline questions are especially challenging, and some chart-reasoning questions require integrating multiple frames.

  • F.3 Alignment Analysis: The analysis defines low-EM/high-BERTScore cases using below-median BLEU-2 and METEOR with at-or-above-median BERTScore, then categorizes four error types.The categories are Grounding Error, Compression, Paraphrase, and Additive Detail.
  • F.3 Alignment Analysis: Animation questions on the Timeline layer, including slowing, speeding up, or pausing visualization, are the most challenging editorial-layer questions for models.Figure 8 illustrates representative animation question-answer pairs across four editorial layers.
  • F.3 Alignment Analysis: Some chart-reasoning aggregation questions require integrating information across two discrete frames rather than answering from a single keyframe.Successful answers require grounding target chart elements across one or multiple frames before numerical calculation.
  • F.3 Alignment Analysis: Most low-EM/high-BERTScore cases involve Grounding Error or Compression, showing semantically similar answers that omit or compress key information and thereby lose exact-match credit.Qwen3.5-9B shows a larger share of Grounding Error, whereas other models show relatively higher compression.
  • G AI Assistants Usage Declaration: AI assistants supported dataset construction, coding, figure design, and language polishing, while authors manually verified and extensively revised all AI-generated outputs.The stated verification process was intended to ensure accuracy and quality.
  • F.3 Alignment Analysis: Representative alignment cases in Table 10 illustrate the different error types through gold answers, model responses, and assigned error labels.The table presents examples corresponding to the qualitative error analysis.
Loading 2608.29711v1…