PaperlayerView exact revision on arXiv

Source-linked AI summary

CRAFT: A Benchmark for Causal Reasoning About Forces and inTeractions

Tayfun Ates, M. Samil Atesoglu, Cagatay Yigit, Ilker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, Deniz Yuret

arXiv:2012.04293v3cs.AIcs.CLcs.CV

TL;DR

CRAFT addresses the challenge of evaluating whether models can reason about physical actions and interactions in complex causal scenes. It introduces a video question-answering dataset with descriptive, counterfactual, and cause/enable/prevent questions; the reported results show that current models cannot solve its questions reliably, unlike human adults.

  • Problem

    Artificial systems remain significantly below human performance on reasoning about physical actions in complex causal scenes.

    Source: §1 · p. 1

  • Method

    CRAFT is a video question-answering dataset with descriptive, counterfactual, and cause, enable, and prevent questions about physical interactions.

    Source: §1 · p. 1

  • Results

    Current models cannot solve CRAFT questions reliably, while human adults can reason about physical interactions between objects.

    Source: §5 · p. 8

  • Takeaways & Limitations

    Developing more effective CRAFT models is a research direction for video QA systems intended to mimic human causal reasoning about forces and interactions.

    Source: §5 · p. 9

    The experiments do not report results for recent neuro-symbolic models such as NS-DR, which require extra object-level annotations.

    Source: §5 · p. 9

Abstract

from arXiv · show

Humans are able to perceive, understand and reason about causal events. Developing models with similar physical and causal understanding capabilities is a long-standing goal of artificial intelligence. As a step towards this direction, we introduce CRAFT, a new video question answering dataset that requires causal reasoning about physical forces and object interactions. It contains 58K video and question pairs that are generated from 10K videos from 20 different virtual environments, containing various objects in motion that interact with each other and the scene. Two question categories in CRAFT include previously studied descriptive and counterfactual questions. Additionally, inspired by the Force Dynamics Theory in cognitive linguistics, we introduce a new causal question category that involves understanding the causal interactions between objects through notions like cause, enable, and prevent. Our results show that even though the questions in CRAFT are easy for humans, the tested baseline models, including existing state-of-the-art methods, do not yet deal with the challenges posed in our benchmark.

1 Introduction

The paper targets the gap between human and artificial reasoning about physical actions in complex causal scenes. It introduces CRAFT, a video question-answering dataset that tests descriptive, counterfactual, and cause/enable/prevent reasoning.

1 Introduction

  • Force Dynamics Theory emphasizes reasoning about multiple forces and how causal language describes patterns of forces in complex scenes.The cited cognitive-science work had focused more on simple causes or event chains than complex causal scenes.Source: §1 · p. 1
  • Existing artificial systems remain significantly below human performance on reasoning about physical actions in complex causal scenes.The paper examines how language and vision interact in reasoning about physical relationships between dynamic objects.Source: §1 · p. 1
  • CRAFT tests causal interactions through cause, enable, and prevent questions alongside descriptive and counterfactual questions.Answering causal questions involves identifying and tracking objects and interpreting their interactions in relation to these categories.Source: §1 · p. 1

2 Related Work

Prior work addresses visual question answering, physical reasoning, and causal cognition, but the related work identifies gaps in combining these capabilities. CRAFT builds on earlier VQA designs while testing force-based causal categories in more temporally complex scenes.

Related VQA datasets

  • Prior hypothetical VQA datasets test physical reasoning, but CRAFT’s scenes are described as temporally more complex.Earlier video question-answering work also explored temporal reasoning and hypothetical consequences of actions or missing objects.Source: §2 · p. 2

Causal reasoning in cognitive science

  • Force Dynamics Theory models cause, enable, and prevent relations between an affector and a patient in complex causal scenes.The related work states that, to the authors’ knowledge, CRAFT is the first VQA attempt to integrate these relationships.Source: §2 · p. 2

Physical reasoning in artificial intelligence

  • Physical-reasoning benchmarks include tasks such as predicting stability or motion, but the cited work notes that their language component is largely missing.The related work also contrasts intuitive-physics VQA datasets, which it says lack visual variations found in PHYRE and Tools.Source: §2 · p. 3

Related VQA datasets

  • CRAFT extends descriptive and counterfactual VQA with questions about causal interactions categorized as cause, enable, and prevent.These categories require learning the semantics of the verbs and the corresponding object interactions and outcomes.Source: §2 · p. 3

3 The CRAFT Dataset

CRAFT generates video-question pairs from varied 2D simulations and uses event graphs, scene states, and functional programs to support questions about temporal and causal reasoning. Its dataset design also varies question language and targets answer-distribution biases.

Dataset scale and splits

  • CRAFT contains approximately 57K video-question pairs from 10K videos, with video-based 60:20:20 splits and easy and hard test settings.The hard setting selects scene layouts not seen during training; the easy setting does not impose that constraint.Source: §3 · p. 3

Video generation

  • Twenty scene layouts generate 10-second, 256 × 256-pixel videos with randomized static elements and dynamic objects affected by gravity and interactions.The static-element configurations vary across scenes, while dynamic-object attributes change until the objects rest.Source: §3 · pp. 3–4, Figure 2 · p. 4

Events

  • The simulation system extracts Start, End, Collision, Touch Start, Touch End, and Enter Basket events, then links events sharing objects in a directed causal graph.Touch events help distinguish collisions from rolling or objects moving together.Source: §3 · p. 4

Simulation representation

  • Each simulation is represented by its initial state, final state, and extracted-event causal graph.Scene states include static and dynamic object attributes such as color, position, and shape.Source: §3 · p. 4

Question generation

  • Functional programs filter events and objects for question generation, and minor object perturbations screen out counterfactual questions whose answers change across trials.The question-generation modules extend the CLEVR approach.Source: §3 · p. 5

Question types

  • The 48 question types span descriptive, counterfactual, and causal categories; some counterfactuals require multiple simulations, while causal questions compare original and counterfactual videos.Descriptive questions include object attributes and temporal analysis, such as counting.Source: §3 · p. 5
  • CRAFT infers object intention from initial velocity to distinguish enabling, causing, and preventing an intended task.An affector enables a task when an intended patient fails without it, causes it when the patient is not intended to act, and prevents it when an intended patient succeeds without it.Source: §3 · p. 5

Language variation and bias reduction

  • Question paraphrases and synonyms add language variety, while answer distributions across simulation-task pairs are kept uniform to discourage identifier-based shortcuts.The generation process aims to keep the overall answer distribution as uniform as possible.Source: §3 · pp. 5–6

4 Experimental Analysis

CRAFT evaluations show that tested models struggle with visual and temporal reasoning in its simulated scenes, especially when test layouts are unseen, while oracle descriptions bring performance close to humans.

4.1 Baselines

  • The baseline suite includes heuristic and text-only systems, image- and video-based LSTM-CNN and MAC variants, adapted TVQA models, and object-centric G-SWM.LSTM-D and BERT-D additionally receive template-based descriptions of simplified causal graphs.Source: §4.1 · pp. 6–7

4.2 Results

  • Descriptive-question accuracy ranges from 23.5%–48.12% in easy and 23.2%–42.9% in hard settings, versus 32.7%–61.4% and 30.1%–56.2% for the other types.The authors suggest the variety of count, shape, and color answers may explain the descriptive-task difficulty.Source: §4.2 · p. 7
  • LSTM-CNN-V does reasonably well on easy splits but generalizes less well on hard splits, and TVQA performs worse than LSTM-CNN-V.TVQA's performance degrades without subtitles or related concept detectors.Source: §4.2 · pp. 7–8
  • Video models typically outperform single-frame methods, while text-only models perform worst; multimodal methods also show a substantial easy–hard gap on CRAFT.The authors attribute the gap to weak generalization to unseen scenes and incomplete detection and localization of physical interactions.Source: §4.2 · p. 7, Table 2 · p. 8
  • MAC variants perform best in the hard setting, while G-SWM performs poorly; the authors suggest many objects may hinder its object-background decomposition.Pretraining G-SWM before a two-stage framework is left as future work.Source: §4.2 · p. 8
  • In the hard setting, human subjects exceed neural baselines by more than 29%, while oracle-description models outperform tested neural models and BERT-D slightly exceeds humans on descriptive questions.The human study selected 56 participants who answered at least 75% of the questions.Source: §4.2 · p. 8

5 Conclusion

CRAFT tests physical and causal reasoning through video questions, and the reported results show that humans can reason about object interactions while current models cannot answer reliably. The appendices document the functional modules and example programs used to represent and answer questions.

5 Conclusion

  • Humans can reason about physical interactions, but current models cannot reliably answer CRAFT questions, leaving substantial room for improvement relative to humans.The conclusion also notes that recent neuro-symbolic models were not evaluated; the approaches discussed require extra object-level annotations.Source: §5 · p. 8

A.1 Functional Modules

  • CRAFT questions use functional programs, with tables listing module input and output types and the full module inventory.The listed module tables cover input, output, object-filter, event-filter, and auxiliary functions.Source: §A.1 · p. 12, Table A.1 · p. 12, Table A.2 · p. 12

A.2 Example Programs

  • Example programs show how simulation-environment functions extract correct answers for descriptive, counterfactual, cause, enable, and prevent questions.The appendix provides one or more functional-program examples for each of these question categories.Source: §A.2 · p. 15, Figure A.1 · p. 15, Figure A.2 · p. 16

A.3 Implementation Details

The implementation details specify baseline training schedules, video preprocessing, and shared modeling choices, with validation performance used for tuning.

A.3 Implementation Details

  • LSTM and single-frame models train for 75 epochs at batch size 64, while temporal baselines train for 30 epochs at batch size 32.G-SWM trains for 100 epochs at batch size 64 with Adam and a 0.0001 learning rate.Source: §A.3 · p. 18
  • Videos are sampled at 5 fps and resized to 112 × 112 pixels; word embeddings are randomly initialized at length 256.Pretrained convolutional video and image encoders are jointly trained with the rest of each network.Source: §A.3 · p. 18
  • Models use negative log-likelihood over possible answers and are tuned on validation performance; TVQA+ uses full precision while other baselines use mixed precision.Single-frame training takes 2 minutes, and video-model training takes approximately 20–30 minutes per epoch.Source: §A.3 · p. 18

A.4 Detailed Quantitative Results

The appendix organizes quantitative results by scene and question type across easy and hard test splits, and supplies additional examples of questions and model predictions.

A.4 Detailed Quantitative Results

  • Tables A.8 and A.9 report tested-model performance by scene on the easy and hard CRAFT test splits, respectively.The supplied captions identify each table’s scope but do not provide individual performance values.Source: §A.4 · p. 18, Table A.8 · p. 19, Table A.9 · p. 19
  • Tables A.10 and A.11 report tested-model performance by question type on the easy and hard test splits, respectively.The appendix also describes question subcategories and gives a sample question in Table A.7.Source: §A.4 · p. 18, Table A.7 · p. 18, Table A.10 · p. 20

A.5 Additional Examples

  • Additional examples pair sample CRAFT questions with oracle descriptions and baseline predictions, with descriptive examples on the left and other categories on the right.The examples include predictions across question categories.Source: §A.5 · p. 21, Figure A.6 · p. 21

A.6 Human Evaluation

The human evaluation used an online study in which participants watched videos and answered multiple-choice questions; participation was voluntary and approved by the university’s IRB.

A.6 Human Evaluation

  • Participants completed the human evaluation online through Qualtrics, with an estimated completion time of 20–30 minutes.They received no bonus or wage and participated voluntarily.Source: §A.6 · p. 21
  • The study collected no personal identifying information, and participants could leave whenever they wanted.The study reported no expected negative outcomes for participants.Source: §A.6 · p. 21
  • Koç University’s Institutional Review Board approved the study under Protocol no: 2021.164.IRB3.073.Participants saw videos and multiple-choice questions; the instruction page is shown in Figure A.7.Source: §A.6 · p. 21, Figure A.7 · p. 22

B Datasheet for CRAFT

CRAFT was created to support research on the gap between human and artificial systems in reasoning about physical relationships. The dataset documentation follows the Datasheets for Datasets guideline.

  • The dataset documentation follows the guideline in Datasheets for Datasets.Source: B Datasheet for CRAFT · p. 23
  • CRAFT was created to facilitate research on closing the gap between human and artificial systems’ understanding of physical relationships.Source: B Datasheet for CRAFT · p. 23
  • Researchers from Hacettepe University and Koç University created CRAFT.The listed creators are Tayfun Ates, M. Samil Atesoglu, Cagatay Yigit, Erkut Erdem, Ilker Kesen, Mert Kobas, Aykut Erdem, Tilbe Goksun, and Deniz Yuret.Source: B Datasheet for CRAFT · p. 23

What do the instances that comprise the dataset

CRAFT combines videos and questions with structured information that supports answering and analyzing questions about dynamic scenes. The dataset also includes variations created by removing objects.

Are there recommended data splits (e.g., training, development/validation, testing)?

CRAFT provides easy and hard data-split settings, with the hard setting testing on unseen scene layouts. Its instances are automatically generated, and the documentation reports no identifying or sensitive information.

Are there recommended data splits (e.g., training, development/validation, testing)?

  • In the hard setting, train, validation, and test samples use 12, 4, and 4 layouts, respectively, so test layouts are unseen during training.Both settings have non-overlapping train, validation, and test sets; the easy setting may include all layouts in each split.Source: Are there recommended data splits… · p. 24

Does the dataset identify any subpopulations (e.g., by age, gender)?

If the dataset is a sample, then what is the population?

CRAFT’s preprocessing filters questions for answer stability under small perturbations and reduces dominant answers. The dataset can support additional tasks and is publicly distributed, with extensions planned.

If the dataset is a sample, then what is the population?

  • The authors frame CRAFT as a sample dataset for research on closing the human–artificial-systems gap and describe scripts for expanding its objects, layouts, and question types.Source: If the dataset is a sample, then what… · p. 25
  • Two preprocessing filters remove question-video pairs whose answers change under small perturbations and pairs with dominant answers.The authors report that the filters address answer inconsistency that troubled human subjects and aim to reduce model shortcuts from frequent answers.Source: If the dataset is a sample, then what… · p. 25

What (other) tasks could the dataset be used

When will the dataset be distributed?

Will the dataset be updated (e.g., to correct

Related papers

Loading 2012.04293v3…