DataVista: Diagnosing Multimodal LLMs on Data Video Understanding

1 INTRODUCTION

• Existing general-video and static-chart benchmarks do not systematically assess evidence localization, cross-chart integration, and narrative understanding within continuous videos.Data videos combine animated visualizations with narration and scene transitions to convey data-driven stories over time.Source: §1 · p. 1
• The framework progresses from data perception to temporal reasoning and narrative understanding, spanning 10 question types.These dimensions cover reading values and visual elements, integrating information across time and charts, and interpreting design intent and argument organization.Source: §1 · p. 2
• Gemini-3.1-Pro reaches 70.0% accuracy, yet models are weakest on Causal Reasoning and Narrative Structure.The evaluation covers 19 mainstream MLLMs; increases in frame counts and subtitles yield relatively limited gains in narrative understanding.Source: §1 · p. 2
• DATAVISTA is presented as the first dedicated benchmark for data video understanding, containing 961 real-world data videos spanning five topic domains and paired with 6,775 multiple-choice evaluation questions.It is used to systematically evaluate 19 mainstream MLLMs.Source: §1 · p. 2

Video understanding benchmarks

• General-video benchmarks assess multimodal, temporal, and long-video understanding, while DATAVISTA targets quantitative evidence in data videos.Its focus is reading dynamic chart information, integrating evidence across charts, and relating that evidence to video claims.Source: §2 · pp. 2–3

Chart understanding benchmarks

• Chart benchmarks extend from individual static charts to multi-chart reasoning, but still use isolated charts or pre-assembled static collections.DATAVISTA instead evaluates reading values from video frames, integrating accumulating evidence over time, and interpreting narrative structure and visual design.Source: §2 · p. 3

Visualization authoring and data-video narrative

• Visualization-authoring systems generate or refine chart artifacts, whereas DATAVISTA evaluates whether models understand charts within dynamic, narrated sequences.Related data-video research has emphasized designing and generating data stories; DATAVISTA examines understanding their data relationships and narrative expression.Source: §2 · p. 3

3 DATAVISTA: DESIGN AND CONSTRUCTION

• The benchmark construction pipeline comprises video collection and filtering, capability hierarchy and question design, then expert annotation and quality control.The pipeline is organized into three stages.Source: §3 · p. 3

3.1 VIDEO COLLECTION AND PREPROCESSING

• Of 5,604 candidate videos, 967 pass screening; the final questions cover 961 videos after review.Screening requires data-driven core information, a sustained visible visualization, and clear communication intent; annotators confirm or correct model classifications. The videos span five domains and 22 subcategories, range from 28 seconds to 15 minutes, and have a median duration of about 5.6 minutes.Source: §3.1 · pp. 3–4

3.2 CAPABILITY HIERARCHY

• The framework organizes questions into three progressive levels and 10 types, moving from chart reading to cross-chart temporal reasoning and narrative interpretation.Data perception requires locating evidence in continuous video before reading it; temporal reasoning integrates evidence across charts and time; narrative understanding addresses story organization, visual intent, counterfactual design, and quality.Source: §3.2 · pp. 4–5

3.3 QUESTION DESIGN

• Question formats include single-answer and multiple-select items, with differentiated distractors designed to test precise reading and discourage shortcuts.Examples include numerically close distractors for data-reading questions, single-chart shortcuts for cross-chart questions, and narrative-grounded options for higher-level questions.Source: §3.3 · p. 5

3.4 EXPERT ANNOTATION AND QUALITY CONTROL

• Five experienced annotators design candidate questions with model-assisted extraction, then watch videos, verify extracted information, and write questions and answers.Four independent reviewers check wording, video evidence, and distractors; disagreements are resolved against the original video, and ambiguous items are removed. The final dataset contains 6,775 questions covering 961 videos.Source: §3.4 · pp. 5–6
• Text-only filtering removes candidates whose predicted answers match references, targeting questions answerable without video evidence.Across five representative models, filtered-out questions are 22.9–32.2 percentage points easier to answer from text alone than retained questions.Source: §3.4 · p. 6

4.1 EXPERIMENTAL SETUP

• The main evaluation tests 19 MLLMs using question stems, answer options, and 50 uniformly sampled frames without audio, with a separate subtitle condition.Exact-match accuracy requires the exact correct answer or option set; human evaluators watch complete videos with original audio.Source: §4.1 · p. 6

4.2 OVERALL PERFORMANCE

• 70.0% overall accuracy for Gemini-3.1-Pro is 21.7 percentage points below the human baseline; GPT-5 reaches 67.4%.Human evaluators achieve 91.7% overall accuracy; models also show their largest reported gap to humans on L3 narrative understanding.Source: §4.2 · pp. 6–7, Table 1 · p. 7

4.3 INPUT EVIDENCE ANALYSIS

• Adding frames raises mean L1 accuracy by 32.0 points, while adding subtitles to frames raises mean L2 accuracy by 6.5 points.Frame gains are smaller on L2 and L3; subtitles added to frames raise L1 and L3 by 2.9 and 2.4 points, respectively. For GPT-5, subtitles raise accuracy on videos longer than 9 minutes from 59.4% to 66.9%, narrowing the gap from the shortest-video group from 8.9 to 1.4 points.Source: §4.3 · p. 8

4.4 FRAME BUDGET SENSITIVITY

• Increasing the frame budget from 8 to 32 raises mean L1 accuracy by 11.6 points, compared with 4.7 on L2 and 2.1 on L3.GPT-5 leads at 8 and 16 frames, while Gemini-3.1-Pro leads at 32 and 50; at 100 frames, L1 can still benefit but L2 and L3 show no consistent improvement.Source: §4.4 · p. 8, Figure 5 · p. 8

4.5 CAPABILITY DIMENSION ANALYSIS

• With frames and subtitles, mean accuracy is 39.2% on Causal Reasoning and 53.9% on Argument Synthesis, versus 73.6% on Cross-Temporal Change.The best model reaches only 44.2% on Causal Reasoning; all five models score between 46.9% and 51.0% on Narrative Structure. Argument Synthesis reaches 88.0% option-level F1 but 53.9% exact match, indicating many partially correct responses.Source: §4.5 · p. 9, Figure 6 · p. 9

4.6 ERROR ANALYSIS

• Among 120 sampled incorrect Gemini-3.1-Pro responses, 47.5% are perceptual errors, 42.5% reasoning errors, and 10.0% instruction-understanding errors.Value and text reading account for 36.7% of errors, while evidence judgment accounts for 37.5%; instruction errors include shifting the specified time range or target object.Source: §4.6 · pp. 9–10

• The evaluated MLLMs remain substantially behind human performance, with persistent difficulty identifying evidence for causal claims and understanding narratives.Source: §5 · p. 10
• DataVista comprises 961 real-world data videos and 6,775 questions across three capability levels and 10 question types.Source: §5 · p. 10
• Gemini-3-Flash assisted with dataset pre-screening and pre-extraction, with outputs checked or corrected by humans and questions written by annotators.GPT-6 Astra assisted with English grammar polishing and wording suggestions; authors reviewed all AI-assisted content and take responsibility for the final paper.Source: AI USE STATEMENT · p. 10
• The benchmark releases team-created annotations for academic research, while source-video copyrights remain with their creators or rightsholders.The data collection and annotation procedures received institutional ethics-board approval, and participating domain experts were fairly compensated.Source: ETHICS STATEMENT · p. 10
• The authors release question annotations and video source records and plan an automated evaluation service and public leaderboard.The planned service is intended to support standardized evaluation and tracking progress in data-video understanding.Source: REPRODUCIBILITY STATEMENT · pp. 10–11

A.2 QUESTION TYPE SPECIFICATION

• Each question type is specified by its definition, target competency, visual evidence requirements, and distractor design, forming the annotation and capability-analysis framework.The specifications cover all ten types, with multiple assessment modes for Data Fact Reading, Cross-Temporal Change, Cross-Chart Comparison, and Narrative Structure.Source: A CAPABILITY HIERARCHY AND QUESTION… · p. 18, §A.2 · p. 19

A.1 DESIGN RATIONALE

• L1 reads chart facts and elements; L2 integrates data relationships and evidence across charts and time; L3 interprets narrative organization and how design communicates information.The hierarchy extends visualization-literacy frameworks to cross-chart relationships and narrative context in data videos.Source: §A.1 · p. 19
• Evidence requirements vary by task: value retrieval uses a chart state, cross-chart comparison combines charts, and animation or presentation-order questions track visual change over time.Multiple frames support aggregation or comparison, while order information supports judgments about change processes and narrative organization.Source: §A.1 · p. 19

Level 1: Data Perception

• L1 covers four data-reading modes—value retrieval, trend identification, extremum detection, and statistical aggregation—and recognition of chart types, labels, units, legends, and color encodings.Evidence may come from one frame or unordered multiple frames for aggregation; distractors can use numerically close values or visually similar chart types.Source: Level 1: Data Perception · pp. 19–20

Level 2: Temporal Reasoning

• L2 includes temporal change tracking and extrapolation, cross-chart visual comparison and arithmetic derivation, causal-evidence identification, and synthesis of key evidence for the video’s argument.These tasks require temporal relationships or evidence from multiple chart segments and narrative stages; distractors target single-time-point or single-chart errors, irrelevant or merely plausible observations, and nonessential supporting facts.Source: Level 2: Temporal Reasoning · pp. 20–21

Level 3: Narrative Understanding

• L3 assesses narrative patterns, data-flow order, cross-section argumentation, visual communication intent, counterfactual design impact, and holistic fidelity, expressiveness, and aesthetics.Its evidence spans multiple narrative stages, design features and context, and—in some tasks—the full video; the example requires identifying a resilient sector across non-adjacent sections.Source: Level 3: Narrative Understanding · pp. 21–22

B.1 VIDEO COLLECTION PIPELINE

• Videos are collected through channel crawling and keyword search, then screened by Gemini-3-Flash and verified or corrected by human annotators.The screening considers video frames, subtitles, and metadata, applying criteria that visualizations be real, central and sustained, and data-driven rather than decorative or narration-dominant.Source: §B.1 · pp. 22–23

B.2 STRUCTURED PRE-EXTRACTION

• Gemini-3-Flash processes each full video into a timestamped evidence index of chart segments, narrative stages, argument chapters, and a reviewer-facing digest for question design.Annotators verify the output segment by segment, correct time ranges and chart descriptions, and add omitted visualizations before using the metadata.Source: §B.2 · pp. 25–26
• The evidence map catalogs every visualization, setting/rising_climax/resolution stages, and 3–8 chart-anchored argument chapters when videos have at least four charts or exceed three minutes.Its digest records chart-family inventory, examinable moments, narration-versus-chart comparison hooks, and stage notes; consistency checks align visualization counts and chart references with declared segments.Source: §B.2 · pp. 25–26

B.3.1 TEXT-ONLY FILTERING

• Of 28,897 screened candidates, 21,804 (75.5%) were filtered out and 7,093 proceeded to human review; across all five models, text-only accuracy was 22.9–32.2 percentage points higher on filtered-out questions.Among candidates retained after text-only filtering, 80.1% passed review, 15.4% passed after manual revision, and 4.5% were rejected.Source: §B.3.1 · pp. 26–27, §B.3.2 · p. 27

B.4 VIDEO SOURCE DISTRIBUTION

• The videos come from professional media, research institutions, educational science-communication channels, and independent data creators, with leading sources varying by topic domain.CNBC contributes 153 Economy videos and Vox 73 Politics videos.Source: §B.4 · p. 27

B.5 TOPIC AND SUBCATEGORY DISTRIBUTION

• DATAVISTA uses five topic domains and 22 subcategories; Economy is largest with 404 videos (42.0%), followed by Culture (173), Politics (152), Society (145), and Science (87).Econ: Market Dashboard (177), Econ: Macro Comparison (115), and Culture: Ranking Race (105) are the largest subcategories.Source: §B.5 · pp. 27–28, Table 4 · p. 28

B.6 VIDEO DURATION DISTRIBUTION

• Video durations have a median of approximately 335 seconds, and about 45.1% fall in the 60–300-second range; views and publication dates also span a broad, recent-skewed collection.The view-count median is approximately 5,349; 68.5% of videos were published in or after June 2025.Source: §B.6 · pp. 28–29, §B.7 · p. 29, §B.8 · pp. 29–30

B.9 CHART TYPE DISTRIBUTION

• Across 961 videos, 18 chart types yield 2,210 entries, while the retained dataset contains 6,775 questions, with most videos carrying five to ten questions.Bar charts appear in at least 64.9% of videos; L1, L2, and L3 account for 2,817, 1,908, and 2,050 questions, respectively.Source: §B.9 · p. 30, §B.10 · p. 30, Table 5 · p. 30

C.1 LEVEL 1: DATA PERCEPTION

• All five representative models miss the 2024 quarterly sales peak for On Holding: Q3 is highest at 754M, but three models select Q4 at 720M.For chart-legend mapping, only Gemini-3.1-Pro correctly identifies the solid light-blue line as the historical lower limit.Source: §C.1 · p. 31

C.2 LEVEL 2: TEMPORAL REASONING

• In argument synthesis, three models omit geographic constraints needed to complete the remittance explanation; in cross-temporal arithmetic, four of five miss the increase, giving 294.85M instead of 274.85M.The arithmetic case requires locating values at two moments before subtracting them.Source: §C.2 · p. 32

C.3 LEVEL 3: NARRATIVE UNDERSTANDING

• Only Gemini-3.1-Pro identifies both typography and side-by-side country juxtaposition as techniques emphasizing demographic statistics.Other models omit the juxtaposition, and some select graphical elements absent from the video.Source: §C.3 · pp. 33–34
• No model fully identifies the counterfactual design effects of moving a Hepatitis B risk table earlier; all miss its potential to increase cognitive load.The intended trade-off combines earlier biological explanation with possible disruption of the opening overview and narrative pacing.Source: §C.3 · p. 34
• Only Gemini-3.1-Pro selects the complete set of evidence-organizing strategies in the House and Senate section; the other models miss either progressive map highlights or the Cook Report grouping.The case concerns how comparison, progressive highlighting, and categorization structure the account.Source: §C.3 · pp. 34–35

D.1 INPUT ORGANIZATION AND PROMPT TEMPLATES

• Evaluation prompts provide 50 uniformly sampled frames, adding timestamped subtitles in the subtitle condition, and require answers, reasoning, and source-linked evidence in a fixed JSON format.The appendix also describes YouTube subtitle sourcing and fallback to another language for 40 videos without an English track.Source: §D.1 · p. 38

D.2 RANDOM SELECTION AND MOST-FREQUENT ANSWERS

• Random baselines use uniform selection among four single-choice options or 15 nonempty multi-select sets; the frequency baseline predicts answers using video-level five-fold cross-validation.All questions from a video remain within one fold.Source: §D.2 · p. 39, Table 6 · p. 39

D.3 STATISTICAL RELIABILITY

• Gemini-3.1-Pro reaches 70.0% overall accuracy versus 31.1% for InternVL3-2B, and its 12.7-point lead over the best open-source model exceeds the reported uncertainty margin.Overall-accuracy interval half-widths are approximately 1.2–1.4 percentage points.Source: §D.3 · pp. 39–40

E.1 DOMAIN PERFORMANCE AND WEIGHTING

• Equal weighting across five domains changes each representative model’s accuracy by at most 0.9 percentage points and preserves their ranking, although domain-level comparisons can differ.In Politics, Claude-Sonnet-4.6 scores 66.0% versus GPT-5’s 62.8%.Source: §E.1 · p. 40, Table 8 · p. 40

E.2 NATIVE VIDEO INPUT COMPARISON

• Complete audiovisual input improves overall accuracy over 50 frames alone by 5.3 points for Pro and 4.8 for Flash, with subtitles recovering some narrated evidence but not chart values absent from narration.Audiovisual input does not improve every response; Flash loses 2.7 points on L2 relative to frames with subtitles.Source: §E.2 · pp. 40–41, Table 9 · p. 40

E.3 TEST-TIME REASONING

• Thinking raises Qwen3-VL overall accuracy from 46.2% to 55.6% for 8B and from 57.3% to 62.9% for 32B, with the largest gains on L2 and inconsistent L3 changes.With subtitles, the strongest open-source result remains 65.5%, below Gemini-3.1-Pro at 71.7%, GPT-5 at 68.8%, and the human score of 91.7%.Source: §E.3 · p. 41, Table 10 · p. 41

E.4 HIGHER FRAME BUDGETS

• Increasing the subtitle-supported frame budget from 50 to 100 raises Qwen’s overall accuracy by 2.6 points but lowers InternVL’s by 0.6; L1 improves for both, unlike L2 and L3.Qwen’s L3 remains at 54.2%, while InternVL declines on L2 and L3.Source: §E.4 · p. 41, Table 11 · p. 42

E.5 MULTI-SELECT DIAGNOSTICS

• Across multi-select types, precision exceeds recall, indicating incomplete coverage of correct options; with subtitles, Argument Synthesis has 96.0% precision, 83.8% recall, and 53.9% exact match.Causal Reasoning has lower F1 than the other four task types in the two- and three-option groups; Argument Synthesis often yields correct selections without a complete answer set.Source: §E.5 · p. 42, Table 12 · p. 42, Table 13 · p. 43

Duration-related declines without subtitles

• Without subtitles, GPT-5 falls from approximately 68% accuracy in the first three duration groups to 59.4% on videos longer than nine minutes, while GPT-4o remains between 54.8% and 57.8%.GPT-5 outperforms GPT-4o in every duration group despite its larger short-to-long gap.Source: §E.6 · p. 43

Subtitles narrow short-to-long performance gaps

• With subtitles, GPT-5’s accuracy on videos longer than nine minutes rises from 59.4% to 66.9%, narrowing its short-to-long gap from 8.9 to 1.4 points.Qwen3-VL-32B rises from 53.1% to 60.4% on the longest videos, near its 60.7% shortest-video accuracy.Source: §E.6 · p. 43
• Subtitle benefits vary by model and duration: Gemini-3.1-Pro gains 4.1 and 9.2 points in the 3–6- and 6–9-minute groups, but 0.3 points for videos longer than nine minutes.Claude-Sonnet-4.6’s short-to-long gap narrows from 5.1 to 1.2 points with subtitles.Source: §E.6 · p. 43

E.7 REPRESENTATIVE ERROR CASES

• The two failures occur when Gemini-3.1-Pro ignores a question constraint despite reading the chart numbers correctly.One shifts the specified time range; the other uses narrated information when only chart evidence is allowed.Source: §E.7 · p. 43
• For the requested January-to-February increase, the model answers with the printed 111.91% one-year change instead of the approximately 88% trough-to-peak gain.The question specifies the trough around January 2026 and peak in February 2026.Source: §E.7 · p. 43, Figure 21 · p. 44
• The model treats a narrated 2034 revenue target of about $2 billion as chart evidence, selecting A, B, D instead of the correct A, B.The chart shows Eli Lilly’s 2.52% and Centessa’s 45.58% intraday gains; the revenue target is spoken, not charted.Source: §E.7 · p. 44, Figure 22 · p. 45

E.8 DIAGNOSTIC ANALYSIS OF AN OPEN-SOURCE MODEL

• The open-source model misreads temporal anchors and chart structure, including treating a 2016 peak as a 2024 value or claiming a zero-start y-axis is truncated.The analysis manually inspected 120 randomly sampled incorrect responses from Qwen3-VL-32B with 50-frame input and subtitles.Source: §E.8 · p. 45
• On causal-reasoning and narrative-argument questions, the model often extracts relevant values and trends but judges whether they support an option incorrectly.The resulting error is a failure to translate locally correct facts into valid support for the overall option.Source: §E.8 · p. 45
• Instruction and mapping errors include choosing a different option than the reasoning supports and mixing projected or other-year values into a year-restricted comparison.The model may correctly derive a roughly 20-fold GDP difference and identify option C in its rationale, yet output D.Source: §E.8 · pp. 45–46
• The findings point to persistent difficulty integrating information coherently and maintaining consistency over long videos.The authors suggest exploring candidate-answer verification and selection.Source: §E.8 · p. 46

E.9 UPLOAD-TIME AND VIEW-COUNT ANALYSIS

• The contamination check compares the 19 evaluated models’ accuracy across videos grouped by upload time and view count.It tests whether older or more-viewed videos show a consistent performance advantage.Source: §E.9 · p. 46
• Gemini-3.1-Pro scores 68.9%, 68.8%, and 71.9% across the three upload-date groups, close to its overall 70.0% accuracy.GPT-5 scores 69.9% on 2026 videos versus 63.2% on pre-2025 videos, a 6.7-percentage-point difference.Source: §E.9 · p. 46
• Accuracy does not increase monotonically with view count; GPT-5 and Gemini-3.1-Pro both score lowest in the most-viewed group.GPT-5 scores 69.1%, 70.7%, 68.4%, and 60.9%; Gemini-3.1-Pro scores 70.7%, 75.7%, 70.6%, and 62.2% across Groups 1–4.Source: §E.9 · p. 46
• Neither analysis shows a systematic performance advantage for older or more widely viewed content.The analysis compares accuracy across upload-date groups and four equal-frequency view-count groups.Source: §E.9 · p. 46

F LIMITATIONS AND FUTURE WORK

• DATAVISTA does not yet cover data video generation or editing, open-ended subjective items, or multilingual comprehension beyond its primarily English videos.The authors cite standardized, reproducible evaluation at scale as the reason for currently relying on multiple-choice questions.Source: F LIMITATIONS AND FUTURE WORK · p. 46
• Future work will add generation and editing evaluation, open-ended items with automated scoring, multilingual datasets, a fine-tuning dataset, and a maintained evaluation website and leaderboard.The proposed platform would incorporate new videos and support evaluation of current and custom models.Source: F LIMITATIONS AND FUTURE WORK · p. 46