Source-linked AI summary
NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions
Junbin Xiao, Xindi Shang, Angela Yao, Tat-Seng Chua
TL;DR
VideoQA models have advanced at describing visible content but remain limited in explaining causal and temporal actions. NExT-QA introduces a manually annotated benchmark with multi-choice and open-ended tasks, and finds that strong performance on descriptive and multi-choice questions does not transfer reliably to causal, temporal, and open-ended answering.
Problem
Existing VideoQA systems can describe visible objects and actions but struggle to reason about their causal and temporal relationships.
Method
NExT-QA provides 5,440 videos and about 52K manually annotated question-answer pairs spanning causal, temporal, and descriptive questions in multi-choice and open-ended formats.
Results
SOTA methods perform well on descriptive questions but remain weak on causal and temporal reasoning, with approximately 10% and 30% gaps for multi-choice and open-ended QA respectively.
Takeaways & Limitations
NExT-QA identifies causal and temporal reasoning, answer generation, and richer video understanding as open challenges for VideoQA research.
Takeaways & Limitations
Open-ended QA remains constrained by both reasoning and language generation, especially for complex questions and longer answers.
Abstract
from arXiv · showhide
We introduce NExT-QA, a rigorously designed video question answering (VideoQA) benchmark to advance video understanding from describing to explaining the temporal actions. Based on the dataset, we set up multi-choice and open-ended QA tasks targeting causal action reasoning, temporal action reasoning, and common scene comprehension. Through extensive analysis of baselines and established VideoQA techniques, we find that top-performing methods excel at shallow scene descriptions but are weak in causal and temporal action reasoning. Furthermore, the models that are effective on multi-choice QA, when adapted to open-ended QA, still struggle in generalizing the answers. This raises doubt on the ability of these models to reason and highlights possibilities for improvement. With detailed results for different question types and heuristic observations for future works, we hope NExT-QA will guide the next generation of VQA research to go beyond superficial scene description towards a deeper understanding of videos. (The dataset and related resources are available at https://github.com/doc-doc/NExT-QA.git)
1. Introduction
NExT-QA targets a gap in VideoQA: models can describe visible objects and actions but struggle to explain causal and temporal relationships. It introduces a manually annotated benchmark with multi-choice and open-ended tasks, revealing persistent weaknesses in deeper reasoning.
- Existing VideoQA models can recognize objects and independent actions but struggle to answer questions about their causal and temporal relationships.
- NExT-QA contains 5,440 videos and about 52K manually annotated question-answer pairs covering causal, temporal, and descriptive questions.
- The benchmark defines multi-choice QA with five candidate answers and open-ended QA requiring short answer generation without candidate options.
- Top-performing methods perform well on descriptive questions but remain unsatisfactory on causal and temporal questions.
- Models effective on multi-choice QA struggle when adapted to open-ended QA, raising concerns about their understanding of causal and temporal action structure.
- The paper contributes a curated benchmark, causal and temporal reasoning tasks, and detailed baseline analyses across question types.
2. Related Work
Prior VideoQA benchmarks often emphasize recognition, short or simulated actions, social behavior, dialogue, or textual plot understanding. NExT-QA instead focuses on causal and temporal action reasoning grounded in real-world video content.
- Early VideoQA datasets commonly generate questions from video descriptions, requiring limited reasoning beyond recognizing objects and actions.
- TGIF-QA uses short GIFs with mostly trivial repetition or transition actions involving a single object.
- Social-IQ addresses causalities and interactions in multimodal social behavior but is small-scale and focused on complex human social behaviors.
- MovieQA and TVQA may include causal or temporal questions, but their reliance on textual plots or dialogue reduces the visual reasoning challenge.
- NExT-QA differs from commonsense-oriented resources by requiring causes and effects to be visible in the video and focusing on causal and temporal action structure.
3. NExT-QA Dataset
NExT-QA is built from realistic videos with operationally defined causal, temporal, and descriptive questions, supporting both candidate selection and answer generation. Its 5,440 videos and 52,044 question-answer pairs emphasize longer, richer reasoning questions, especially causal and temporal ones.
- Question types: Causal questions ask why an action occurs or how an action brings about an observed effect, with both cause and effect visible in the video.
- Question types: Temporal questions test previous, present, or subsequent actions and emphasize interactions among multiple objects.
- Question types: Descriptive questions cover scene locations, objects, attributes, events, binary choices, counting, and other visibly inferable answers.
- Task formats: Multi-choice QA supplies five options, whereas open-ended QA requires models to generate textual answers from the video and question without candidates.
- Construction: The dataset uses 6,000 selected VidOR videos featuring daily-life activities, with train, validation, and test splits in a 7:1:2 ratio.
- Construction: Annotation involved three stages over one year and 100 undergraduate annotators, with separate question and answer annotation and quality supervision.
- Construction: Post-processing yielded 5,440 valid videos and 52,044 question-answer pairs after balancing yes/no answers and removing large-count questions.
- Construction: Multi-choice distractors were retrieved from similar questions, filtered for semantic similarity, sampled four at a time, and manually checked.
4. Experiments
The experiments evaluate diagnostic baselines and established VideoQA models across causal, temporal, and descriptive questions, using multi-choice and open-ended metrics. Results show that semantic or answer-only shortcuts are insufficient, while stronger video reasoning models still struggle particularly with causal and temporal reasoning and open-ended generation.
- Evaluation: Multi-choice QA reports accuracy, while open-ended QA uses WUPS, with accuracy retained for binary and counting descriptive questions.WUPS evaluates generated answers after stop-word removal and lemmatization, incorporating semantic similarity.
- Baselines: Random selection achieves about 20% accuracy because correct answers are evenly distributed among five options.This serves as a basic diagnostic baseline for the multi-choice task.
- Baselines: Popularity-based guessing improves descriptive questions but is only slightly better for causal questions and worse for temporal questions.The result is consistent with descriptive questions having more frequent answers, whereas guessing is ineffective for reasoning-oriented questions.
- Baselines: Answer-similarity baselines perform poorly, indicating that questions cannot be answered reliably from semantic similarity between questions and answers alone.SimAA is worst overall, while SimQA improves on causal and temporal questions but remains limited on descriptive questions.
- Established VideoQA Models: Established VideoQA methods use varied architectures and visual reasoning techniques, including recurrent encoders, attention, memory modules, self-attention, and hierarchical conditional relation networks.The benchmark compares methods spanning different video-question encoders and reasoning mechanisms.
- Established VideoQA Models: HCRN shows strong causal and temporal reasoning with GloVe and achieves SOTA results with BERT, but its performance varies with language representation and remains below human performance.Reported gaps include 46.26% versus 87.61% on causal questions, 50.74% versus 88.56% on temporal questions, and 59.33% versus 90.40% on descriptive questions.
- Open-ended QA: Open-ended performance improves less than 3% over BlindQA, mainly because models remain poor on causal and temporal questions.Performance also declines for longer answers, while descriptive-question results are relatively stable as question length increases.
5. Discussion and Conclusion
NExT-QA analysis finds a substantial gap between descriptive performance and causal or temporal action reasoning, especially when models must generate answers. The authors identify graph reasoning, feature integration, contextualization, and language modeling as open directions.
- SOTA methods perform well on descriptive questions but remain weak in causal and temporal action reasoning, with gaps of approximately 10% in multi-choice and 30% in open-ended QA.
- Graph models such as HGA are reported as superior for causal and temporal relation reasoning and as a promising research direction.
- Naively concatenating appearance and motion features usually produces sub-optimal results, motivating better methods for modeling and merging them.
- Fine-tuned BERT shows absolute superiority on causal and temporal questions, whereas pre-trained BERT representations generalize worse than GloVe because of TextQA bias.
- Models effective on multi-choice QA struggle to answer open-ended questions automatically, raising concern about their understanding of causal and temporal action structures.
- Open-ended QA is constrained by both reasoning and language generation, including weak understanding of complex questions and generation of longer answers.
A.1. Data Statistics
NExT-QA emphasizes questions about explaining temporal actions rather than only identifying scene elements. Its questions mostly use why, how, and what formulations involving actions.
- Most NExT-QA questions ask “why did/does ...” or “how/what did/does ...”.
- NExT-QA shifts VideoQA attention from scene-recognition questions toward explanations of temporal actions.
- The dataset’s rich causal and temporal questions make it a distinctive resource for video understanding.
A.2. Dataset Comparison
NExT-QA contains relatively long questions and is presented as a dataset designed to support deeper video understanding beyond description. Its question distribution is illustrated through comparisons and figures.
- NExT-QA questions average 12 words, whereas MSVD-QA and MSRVTT-QA questions are mostly 5 words and TGIF-QA and ActivityNet-QA questions are mostly about 8 words.
- Statistical comparisons position NExT-QA as opening challenges and opportunities for understanding video content beyond description.
- Figure 7 reports the distribution of NExT-QA questions by their first three words, excluding “the” from each question.
- Annotated examples in the open-ended and multi-choice formats indicate that answers can be visually inferred from the video content.
B.1. Evaluation
NExT-QA evaluates generated answers with WUPS, a soft accuracy measure that compares predicted and reference answer tokens using semantic similarity. Wu-Palmer similarity supplies the token-level comparison.
- WUPS evaluates generated answers by comparing predicted and reference token sets while accounting for synonyms and semantic similarity.
- Wu-Palmer similarity assigns higher scores when words share more common ancestors and occupy similar depths in a taxonomy.
- The Wu-Palmer calculation uses the least common ancestor and the depths of the two compared words.
B.2. Answer Decoders
The decoder comparison shows that architecture choice affects question-type performance: attention-based decoders improve causal reasoning and overall results, while other designs specialize in descriptive or temporal questions.
- Experimental setup: The decoder study compares multiple answer-decoder architectures, with results reported in Table 8 using HGA on the validation set.
- Decoder comparison: AttVid and AttQns achieve better causal-question performance and consequently better overall results than the other decoder approaches.The authors attribute the strength of these decoders to their causal-question performance.
- Decoder comparison: naiveTrans performs best on descriptive questions but still struggles with causal and temporal questions.
- Decoder comparison: QnsAns performs better on temporal questions but remains weak on causal questions, keeping its overall WUPS score low.
C. Results Analysis and Discussion
NExT-QA exposes a gap between shallow video description and deeper causal or temporal reasoning, especially when models must generate answers rather than select them. Its analyses show that current systems can sometimes produce semantically reasonable answers, yet remain limited by video comprehension and language generation.
- Results analysis: Causal and temporal questions require deeper understanding of videos and questions than shallow descriptive questions, and current models remain weak on them.
- Results analysis: Models effective on multi-choice QA usually fail to answer open-ended questions correctly, especially for causal and temporal questions.
- Results analysis: Some open-ended predictions are semantically reasonable despite receiving low WUPS scores relative to the reference answers.
- Results analysis: Some predictions are semantically reasonable but irrelevant to the video contents, indicating difficulty with video comprehension or language generation.
- Benchmark scope: NExT-QA benchmarks causal and temporal action reasoning in videos rich in object interactions from real-daily activities.
- Qualitative analysis: Figure 12 visualizes predictions for multi-choice and open-ended QA, while Figure 13 presents five-option multi-choice examples.