Source-linked AI summary
CLEVRER: CoLlision Events for Video REpresentation and Reasoning
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, Joshua B. Tenenbaum
TL;DR
Existing video reasoning benchmarks emphasize perception and pattern recognition, leaving temporal and causal structure less directly evaluated. CLEVRER addresses this gap with a controlled collision-video dataset covering four reasoning types and evaluates models that combine visual, dynamical, causal, and symbolic components. Baseline models perform well on descriptive questions but poorly on causal tasks, while the proposed NS-DR oracle explicitly joins these components.
Problem
Video reasoning benchmarks have largely emphasized complex visual and language inputs rather than temporal and causal structure.
Method
CLEVRER is a controlled diagnostic dataset of collision videos with four question types and symbolic representations of motion, events, and question logic.
Results
Baseline models perform well on descriptive questions but struggle on explanatory, predictive, and counterfactual causal questions; NS-DR explicitly combines perception, dynamics, causal relations, and symbolic logic.
Takeaways & Limitations
The evaluation suggests that causal video reasoning requires object-centric representations together with explicit modeling of dynamics, causal relations, and symbolic question logic.
Takeaways & Limitations
CLEVRER and NS-DR are preliminary steps toward more realistic video reasoning setups and do not yet capture true intelligence beyond pattern recognition.
Abstract
from arXiv · showhide
The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER), a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human casual judgment, CLEVRER includes four types of questions: descriptive (e.g., "what color"), explanatory ("what is responsible for"), predictive ("what will happen next"), and counterfactual ("what if"). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.
1 INTRODUCTION
CLEVRER addresses temporal and causal reasoning in videos by simplifying visual recognition while emphasizing interactions among objects and events. It introduces a controlled benchmark spanning descriptive, explanatory, predictive, and counterfactual questions, and evaluates existing models plus an oracle neuro-symbolic model.
- CLEVRER studies temporal and causal reasoning in videos while simplifying visual recognition and emphasizing interacting objects’ complex structure.
- The benchmark uses a fully controlled environment designed to isolate logical reasoning with simple visual scenes and language and provide effective model diagnostics.
- CLEVRER contains 20,000 synthetic collision videos, over 300,000 questions and answers, ground-truth motion traces, event histories, and functional programs.
- Its questions cover descriptive, explanatory, predictive, and counterfactual reasoning about videos.
- State-of-the-art models perform well descriptively but struggle with explanatory, predictive, and counterfactual questions; NS-DR explicitly combines perception, dynamics and causal modeling, and symbolic logic.
2 RELATED WORK
The related work situates CLEVRER at the intersection of video understanding, visual question answering, and visual reasoning. Existing approaches span attention, compositional reasoning, and video-language modeling, while CLEVRER adds controlled temporal and causal evaluation.
- CLEVRER is positioned within video understanding, visual question answering, and physical and causal reasoning research.
- Prior video research includes captioning, natural-language video segment localization, and video question answering using large-scale video datasets.
- CLEVRER compares with image and video benchmarks as a controlled, well-annotated dataset covering description, explanation, prediction, and counterfactual reasoning.
- Visual reasoning models include hierarchical attention, joint attention, and compositional reasoning approaches such as SAN, MAC, and related systems.
3 THE CLEVRER DATASET
CLEVRER uses physics-engine videos of colliding tabletop objects paired with machine-generated questions and executable functional programs. Its controlled annotations expose temporal, causal, and compositional reasoning requirements across four question types.
- 3.1 VIDEOS: CLEVRER videos depict object motion and collisions on a flat tabletop simulated by a physics engine and rendered by a graphs engine.
- 3.1 VIDEOS: The dataset provides ground-truth motion traces and histories for all objects and events, enabling diagnostics of temporal and causal reasoning.
- 3.2 QUESTIONS: Each multiple-choice option can combine with a question to form a joint logic trace for program-based reasoning.
- 3.1 VIDEOS: Objects have compositional attributes, while enter, exit, and collision events have fixed participant counts and form an abstract video representation.
- 3.1 VIDEOS: Causal structures define events through object participation and preceding events on the same object, supporting responsibility and counterfactual reasoning.
- 3.2 QUESTIONS: CLEVRER pairs each video with machine-generated descriptive, explanatory, predictive, and counterfactual questions, each linked to an executable functional program.
4 BASELINE EVALUATION
Baseline evaluation on CLEVRER compares language-only, video question-answering, and compositional visual-reasoning models across descriptive and causal tasks. Models perform better when they use compositional or object-aware representations, but most still struggle with causal reasoning, motivating explicit dynamics and causal modeling.
- Baseline models: Baseline evaluation covers language-only, video question-answering, and compositional visual-reasoning model families on CLEVRER.Descriptive questions are treated as multiclass classification, while multiple-choice questions use binary classification for each question-choice pair.
- Baseline models: 1000 programs train the IEP (V) and TbD-net (V) program generators in the reported baseline comparison.All models are trained on the full training set.
- Model details: The proposed model combines a video frame parser, question parser, dynamics predictor, and symbolic program executor.Together, these components produce object-based frame representations, functional programs, predicted dynamic scenes, and answers.
- Descriptive reasoning: Compositional reasoning models achieve more competitive descriptive performance than video question-answering models, which remain limited by question and visual-context compositionality.The comparison includes TbD-net and MAC against video QA models such as TVQA+.
- Causal reasoning: Most baselines perform poorly on explanatory, predictive, and counterfactual questions, while MAC (V) and TbD-net (V) provide only marginal gains over random and language-only baselines.These causal tasks require understanding beyond perception-based reasoning.
- Causal reasoning: Object-aware representations improve causal-task performance: TVQA+ performs strongly on prediction, and MAC (V+) improves over MAC (V) across all tasks.The evaluation identifies object recognition, dynamics and causal-relation modeling, and symbolic question logic as essential elements.
5 NEURO-SYMBOLIC DYNAMIC REASONING
NS-DR combines object-centric perception, learned dynamics, question parsing, and symbolic program execution for temporal and causal video reasoning. Its evaluation uses four question types and compares the full model with a no-events variation.
- Model architecture: NS-DR combines a video frame parser, PropNet dynamics predictor, attention-based question parser, and symbolic program executor.The parser extracts object-centric representations, PropNet models dynamics, the question parser produces functional programs, and the executor runs them on predicted traces.
- Model architecture: The video frame parser detects objects, attributes, masks, and confidence scores from each input video frame.It uses Mask R-CNN with a ResNet-50 FPN backbone and is trained on annotated video frames.
- Model architecture: The program executor filters objects and events by attributes, motion states, temporal order, or causal relations before returning an answer.Its modules operate over motion and event traces extracted by the dynamics predictor.
- Results: NS-DR achieves 88.1% accuracy on descriptive questions when its question parser is trained with 1,000 programs.The reported evaluation covers all four CLEVRER question types and also includes an NS-DR (NE) variation without explicit events.
- Results: NS-DR (NE), which uses predicted motion traces and identifies collisions from velocity changes, performs comparably to the full model.The comparison tests whether explicit event representations are necessary for the reported system.
- Interpretation: The model’s design uses symbolic representations as a common ground for vision, language, dynamics, and causality.This representation is intended to capture compositional structure in both video causal relations and question logic.
6 CONCLUSION
The paper presents CLEVRER and related models as preliminary steps toward systematic temporal and causal video reasoning. It argues that future video understanding should model dynamics and causal relations rather than rely only on passive knowledge extraction.
- Conclusion: The paper introduces benchmark tasks intended to facilitate research on temporal and causal reasoning in videos.It frames this area as a challenging problem that has only recently begun receiving attention from modern AI systems.
- Conclusion: CLEVRER and the NS-DR model are preliminary steps toward video understanding that captures dynamics and causal relations.The conclusion presents both as early contributions rather than a completed solution.
- Implications: The paper connects internal understanding of dynamics and causal relations to applications such as dynamic robot manipulation under complex causal conditions.This application is offered as an example of why video reasoning should go beyond passive knowledge extraction.
A DESCRIPTIVE QUESTION SUB-TYPES AND STATISTICS
CLEVRER descriptive questions are divided into five sub-types, with answer frequencies balanced within each subtype to reduce bias. Figure 6 reports the resulting subtype and answer-space distributions.
- Descriptive question sub-types: Descriptive questions in CLEVRER consist of five sub-types.The sub-types define the descriptive-question breakdown shown in Figure 6.
- Answer-space statistics: Answer frequencies are balanced within each descriptive-question subtype to reduce bias.Figure 6 shows the resulting answer-space distribution for these sub-types.
B NS-DR MODEL DETAILS AND TRAINING PARADIGM
NS-DR builds an object- and relation-centric video representation, propagates interactions through a neural dynamics predictor, and rolls the scene forward under altered conditions.
- Perception module: The frame parser provides object-centric representations from video frames using object masks, intrinsic attributes, and confidence scores.It is implemented as a Mask R-CNN with a ResNet-50 FPN backbone.
- Neural dynamics predictor: A relation predictor determines whether two objects will collide at the next time step.
- Neural dynamics predictor: NS-DR encodes objects and relations, then iteratively propagates relation effects to receiver objects through object and relation propagators.The relation set N_i contains relations for which object i is the receiver.
- Training: The predictor is trained by minimizing the L2 distance between predicted and real future object states and relations using stochastic gradient descent.
- Rollout representation: Its output contains object states and relations across observed and rollout frames, enabling motion and event traces under different rollout conditions.Removing an object is represented by erasing its corresponding vertex and associated structure.
C EXTRA EXAMPLES FROM CLEVRER
The paper provides additional CLEVRER examples, including sample videos, questions, and functional programs for descriptive, explanatory, predictive, and counterfactual reasoning.
- Dataset release: The paper states that the full CLEVRER dataset will be made available for download.
- Extra examples: Figure 8 shows sample CLEVRER videos and questions with stroboscopic imaging used to visualize motion.
- Program executor: Tables 4–6 document the functional modules and input/output data types of NS-DR’s program executor.
- Question programs: Figures 9–11 visualize functional question programs for descriptive, explanatory, predictive, and counterfactual questions.Figure 9 covers descriptive programs, Figure 10 explanatory programs, and Figure 11 predictive and counterfactual programs.