FunQA: Towards Surprising Video Comprehension

• FunQA targets a gap in VideoQA benchmarks, which largely use ordinary videos and leave visually grounded understanding of humor, creativity, and illusions underexplored.Source: §1 · pp. 2–3
• The benchmark contains 4.3K surprising videos and 312K manually annotated free-text QA pairs, with responses averaging 34.2 words rather than NExT-QA’s 2.6 words.Source: §1 · p. 3
• FunQA covers HumorQA, CreativeQA, and MagicQA, whose videos feature unexpected contrasts, intriguing disguises, and seemingly impossible performances.Source: §1 · p. 3
• Its tasks assess counter-intuitive timestamp localization, detailed description, reasoning about surprise, and higher-level title generation.Source: §1 · p. 4
• FunMentor refines VLM answers through detailed multi-turn dialogues, using targeted prompts to improve fluent, logical, and persuasive explanations of surprising content.Source: §1 · p. 4

• Most VideoQA datasets use human-centric, game, synthetic, or egocentric videos, but commonly emphasize ordinary content and limited reasoning formats.Source: §2 · pp. 4–5
• MovieQA and TVQA rely heavily on dialogue and plot summaries, limiting their challenge to visual reasoning.Source: §2 · p. 5
• TGIF-QA uses mostly three-second GIFs and focuses on action description, restricting evaluation of complex reasoning.Source: §2 · p. 5
• NExT-QA introduces open-ended questions but mainly covers daily-life videos with short answers, while FunQA-related humor datasets often depend on audio and narrative cues.Source: §2 · p. 5
• VideoQA methods progressed from LSTMs and graph networks to Transformers and VLMs, but many approaches were developed for short videos and standard comprehension settings.Source: §2 · p. 5

3.1 Task Definition

• FunQA defines four task groups: counter-intuitive timestamp localization, detailed description, counter-intuitiveness reasoning, and higher-level tasks such as titles, creativity scores, and magic explanations.Source: §3.1 · pp. 6–7

3.2 Dataset Statistics

• The dataset contains 4,365 clips, 311,950 QA pairs, 23.9 hours of video, and free-text description or reasoning answers averaging 34.24 words.Source: §3.2 · p. 7
• More than 90% of annotations show high consensus and only 1% show low consensus, supporting the reported objectivity and reliability of FunQA.Source: §3.2 · p. 7, Table 2 · p. 8

3.3 Dataset Construction Pipeline

• The construction pipeline crawled, manually cleaned, annotated, reviewed, translated, and expanded videos, producing English, multilingual, multiple-choice, dialogue, and counter-intuitive-reasoning variants.Source: §3.3 · pp. 7–8

3.4 Quality Control

• Quality control combined multi-person review, a secondary 10% annotation check, consensus evaluation, and a human study testing whether eight static frames suffice for understanding.Source: §3.4 · pp. 8–9, Table 3 · p. 8

• FunMentor first collects factual video information, evaluates fluency, accuracy, logic, and humorous insight, then sends tailored feedback for answer revision.Source: §4 · p. 9

5.1 Model Zoo

• The experiments evaluate caption-based and instruction-based model families with BLEURT and GPT-4 for free-text tasks, while timestamp localization uses IOU.Table 4 covers localization, description, reasoning, title generation, and creativity scoring; traditional BLEU-4 and CIDEr scores are close to zero.Source: §5.1 · p. 10, §5.2 · p. 10, Table 4 · p. 11

5.3 Results and Observations

• Caption-based models emphasize description and often omit temporal information, whereas instruction-based models reason more effectively but tend to underperform on description tasks.The comparison is qualitative because matching words in reasoning responses can inflate scores without preserving meaning.Source: §5.3 · pp. 10–12
• Timestamp localization is the most challenging task: current VLMs could not solve H1, C1, or M1 because they lack temporal awareness.The study adds Random Guess, Mean Time Span, and TimeChat baselines to analyze this task.Source: §5.3 · p. 11, Table 5 · p. 11
• FunMentor improves Video-ChatGPT substantially on H2 and H4, although gains on H3 reasoning are relatively modest.The reported improvements support agent-based refinement, but counter-intuitive reasoning remains challenging.Source: §5.3 · p. 13

5.4 Comparison with Previous Benchmarks

• FunQA is harder than NExT-QA for current systems: GPT-4 without video resembles random guessing on FunQA-MC, and GPT-4V falls to 39 on FunQA-MC-R.FunQA-MC-R specifically emphasizes counter-intuitive reasoning, highlighting difficulty beyond conventional inferential questions.Source: §5.4 · p. 14, Table 8 · p. 14

6 Limitations and Future Work

• The dataset primarily contains video-level data and annotations, leaving spatial, temporal, and object-level detail as a future enhancement.The authors suggest denser annotations akin to PVSG.Source: §6 · p. 14
• Chinese-to-English translation may retain cultural differences despite GPT-based refinement and manual correction.The authors specifically note that language differences can persist after translation and refinement.Source: §6 · p. 14
• Future work aims to enrich annotations, develop more accurate open-ended evaluation metrics, and steer models toward deeper video reasoning.Source: §6 · p. 14
• Future research is advised not to use the FunQA testing set to preserve fair comparisons and prevent data leakage.Source: §6 · p. 15

A.1 Dataset Construction Pipeline

• FunQA collects 4,365 videos from HumorQA, CreativeQA, and MagicQA to emphasize visual, counter-intuitive, and spatial-temporal reasoning.HumorQA contains 1,769 videos, CreativeQA 927, and MagicQA 1,672.Source: §A.1 · p. 20

A.3 Data Statistics

• The appendix provides benchmark comparisons, raw-data statistics, pre-extension statistics, and annotation word-cloud visualizations.Source: Table A1 · p. 21, Table A2 · p. 24, §A.3 · p. 24

A.1 Dataset Construction Pipeline

• Annotations cover counter-intuitive timestamp localization, objective description, and reasoning about why each video is interesting or counter-intuitive.Magic-method annotations were limited because nonprofessional annotators struggled to provide complete explanations.Source: §A.1 · p. 21
• Videos are filtered for fun moments and sensitive content, with trained annotators and multi-person, multi-round reviews supporting annotation quality.Every released annotation was reviewed by three annotators, and 10% of videos received a consistency evaluation.Source: §A.1 · pp. 22–23

A.2 Language Augmentation

• GPT-3.5 translates Chinese answers into English and generates five synonymous variants, while incorrect or culturally ambiguous outputs are filtered or manually corrected.Cultural translation issues occur especially in the Title Task.Source: §A.2 · p. 23, Figure A4 · p. 24

B FunQA Extension Datasets

• GPT-3.5 expands high-quality FunQA annotations tens of times, supporting scalable dataset extensions.Source: B FunQA Extension Datasets · p. 24
• Manual screening and modification address GPT-3.5 translation ambiguities caused by Chinese homophones and culture-specific meanings, especially for video titles.Source: Figure A5 · p. 25

B.1 FunQA-MC Dataset

• FunQA-MC converts QA pairs into multiple-choice questions with short answers, while FunQA-MC-R samples 100 counter-intuitive reasoning QA pairs.FunQA-MC is designed for arbitrary models and focuses on description questions.Source: §B.1 · p. 25

B.2 FunQA-DIA Dataset

• FunQA-DIA converts QA pairs into recursive dialogues with added context and freely generated future dialogue turns.The design targets dialogue-form LLM inputs and expands the available information.Source: §B.2 · p. 25

C.1 Real Fact Question

• FunMentor first asks VLM-answerable factual questions to collect objective information about each video.Source: §C.1 · p. 28, Figure A9 · p. 28

C.2 Prompts Design

• FunMentor then uses predefined multi-round dialogues with increasingly precise instructions to guide VLMs toward correct answers.Source: §C.2 · p. 28

D More Details of Experiment

• The appendix supplies the complete benchmark results with traditional metric scores.Source: D More Details of Experiment · p. 28

D.1 Significance of New GPT-4 Based Metric

• GPT-4 evaluates descriptions, counter-intuitiveness explanations, and titles through prompts tailored to content, logic, common sense, detail, and language quality.Separate prompt designs target detailed description, counter-intuitiveness reasoning, and title evaluation.Source: §D.1 · pp. 28–29
• WUPS can assign a high score to semantically opposite words and cannot evaluate phrase- or sentence-level answers reliably.For example, predicting “black” instead of “white” can receive a WUPS score of 0.92.Source: §D.1 · p. 31
• Traditional metrics struggle with complex logical text, whereas GPT-4 first understands the text and then evaluates similarity through prompt design.Source: §D.1 · p. 31
• GPT-4 can produce different evaluations for identical content, so the authors average multiple evaluations in such cases.Figure A15 illustrates this instability across conversations.Source: §D.1 · p. 34

D.2 On Failure of the Timestamp Localization Task

• No evaluated model successfully answers the timestamp localization task; models describe video content instead of identifying the requested time period.The failure appears in answers expressed in frames or seconds.Source: §D.2 · p. 34

D.3 Implementation Details

• The evaluated systems include mPLUG, GIT, VideoChat, Video-ChatGPT, Otter, mPLUG-Owl, and Video-LLaMA, spanning captioning and multimodal instruction approaches.Source: §D.3 · pp. 34, 36
• VideoChat-13B generates multimodal descriptions with timestamps using QFormer video embeddings and LLAMA-13B, while the experiment uses eight video segments.Source: §D.3 · p. 34
• GPT-4 can return different judgments for the same content across conversations, a property relevant when interpreting its evaluation outputs.Source: Figure A15 · p. 35
• Video-LLaMA represents videos and audio through separate vision-language and audio-language branches with frame and audio-segment embedding layers.Source: §D.3 · p. 36

D.4 Ablation Experiment about Audio

• Muting audio leaves traditional scores essentially unchanged and changes GPT-4 scores only slightly, supporting FunQA’s visual-centric challenge.The authors attribute the small GPT-4 variation mainly to evaluation-method instability.Source: §D.4 · p. 36, Table A5 · p. 37

D.5 More Examples

• The human-performance study samples 100 HumorQA pairs and has five people answer H2, H3, and H4 for comparison with benchmark scores.Source: §D.5 · p. 37, Table A6 · p. 37

D.6 More Examples

• CreativeQA and MagicQA examples show models missing key objects, creative explanations, and the visual effects underlying the videos.Only VideoChat identifies the bicycle in the CreativeQA example, while all three models largely miss the magic effect in the MagicQA example.Source: §D.6 · p. 37

E.1 (Potential) More Essential Factors

• Models often detect objects but fail to understand contextual relationships between sequential events, motivating deeper investigation of video description.Source: §E.1 · p. 37
• Understanding counter-intuitive videos requires common-sense reasoning about normal events, and humor may also depend on extra knowledge such as humor principles.Source: §E.1 · p. 39

E.2 Potential Solutions

• Potential solutions include studying model size, data quality, training strategy, and collaboration among multiple models, with efficiency remaining a consideration.Source: §E.2 · p. 39

E.3 The Emphasis of Temporal Dynamics in FunQA

• Temporal order explains the humor of repeated falls: the second person repeats the first person’s mistake, making the sequence itself part of the joke.Source: §E.3 · p. 40
• In CreativeQA, correctly explaining the performance requires linking watering to the sequential growth of the bean-sprout imitation rather than merely detecting sprouts.Source: §E.3 · p. 40