Source-linked AI summary
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
Rohit Girdhar, Deva Ramanan
TL;DR
Video understanding needs benchmarks that distinguish temporal reasoning from scene and object context. CATER provides a synthetic, controllable dataset with compositional actions, occlusion, containment, and diagnostic task variation, and leading models struggle especially with long-term snitch localization. The dataset is intended as an intermediary toward systems that reason over space and time, while remaining complementary to real-world benchmarks.
Problem
Existing video benchmarks can allow frame-based methods to remain competitive because scene and object context may correlate with tasks, despite the need for long-term temporal reasoning.
Method
CATER is a synthetic diagnostic dataset with controllable scene and object factors, compositional-action tasks, and adversarial snitch localization under occlusion and containment.
Results
Leading video models struggle on CATER, especially on snitch localization requiring long-term reasoning; average pooling and short temporal cues perform poorly.
Takeaways & Limitations
CATER provides controllable diagnostics for evaluating whether video architectures reason over space and time rather than relying on short temporal cues or frame averaging.
Takeaways & Limitations
CATER is synthetic and should be used in addition to real-world benchmarks; its demonstrated tasks focus on classification despite richer available annotations.
Abstract
from arXiv · showhide
Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures.
1 INTRODUCTION
Existing video benchmarks often allow frame-based methods to remain competitive because scene and object context can correlate strongly with labels. CATER addresses this by providing controllable, synthetic tasks that require long-term spatiotemporal reasoning.
- Motivation: 2D frame-based models still routinely rank among top video-benchmark performers despite advances in spatiotemporal architectures.This motivates questioning whether averaging predictions over sampled frames is sufficient for video understanding.
- Motivation: Long-term reasoning over temporal order, occlusion, and containment is necessary for inferences that no single video frame can support.The motivating examples include inferring a hidden firearm and tracking a ball beneath moving cups.
- Dataset gap: Existing video datasets can preserve scene and object biases that make actions recognizable from contextual cues rather than temporal structure.The paper identifies strong correlations between benchmark tasks and scene or object context as a limitation of existing datasets.
- CATER: CATER is a synthetic diagnostic dataset for compositional actions and temporal reasoning in dynamic tabletop scenes.It defines primitive action recognition, compositional action recognition, and adversarial target tracking under occlusion and containment.
- CATER: CATER supports scalable dataset generation and controlled diagnostics, and leading video models still struggle on its tasks.The authors vary factors such as occlusion duration and camera motion to analyze model behavior.
2 RELATED WORK
Prior work spans hand-designed and deep spatiotemporal models, diverse video benchmarks, synthetic reasoning datasets, and object-tracking methods. CATER is positioned within these lines of research as a synthetic, diagnostically controlled benchmark for long-term temporal reasoning.
- Spatiotemporal networks: Video action understanding progressed from hand-designed descriptors to sophisticated spatiotemporal deep networks, but video models have struggled to surpass earlier descriptors.The related work contrasts video progress with larger gains reported for image classification and localization.
- Video action understanding datasets: Existing video benchmarks include controlled scripted-action datasets, crowdsourced collections, and diverse videos gathered from movies or online platforms.These datasets differ in collection setting and video diversity.
- Synthetic data: Synthetic computer-vision research includes semantic scene understanding, 3D scene understanding, and complex reasoning benchmarks such as CLEVR.Video-oriented examples include Flash-MNIST, Moving MNIST, and SVQA.
- Synthetic data: Related reasoning benchmarks target causal physical reasoning through question answering, reinforcement learning, prediction, or ranking interfaces.The cited concurrent benchmarks include CLEVRER, PHYRE, COPHY, and IntPhys.
- Object tracking: Object tracking has served as an initial representation for long-term video understanding, with adversarial tracking hiding objects from plain view.Prior applications include determining whether people carry, abandon, or exchange objects.
3 THE CATER DATASET
CATER is a synthetic, controllable video dataset designed to require long-term temporal reasoning while minimizing object and scene bias. It defines progressively complex tasks involving atomic actions, temporal compositions, and the final location of a recursively contained object.
- Dataset design: CATER uses synthetic rendering to control dataset parameters, remove object and scene bias, and support diagnostics across camera motion and object configurations.The dataset provides ground-truth structure for designing multiple video understanding tasks.
- Dataset design: Its object universe extends CLEVR with cubes, spheres, cylinders, cones, a table plane, and a metallic-gold snitch across varied sizes, materials, and colors.Standard objects have three shapes, three sizes, two materials, and eight colors; cones share these attributes.
- Action representation: CATER defines four atomic actions—rotate, pick-place, slide, and contain—with affordances determined by object type.Rotate is available to cubes, cylinders, and the snitch, while pick-place and slide apply to all objects.
- Action representation: Temporal compositions relate action intervals using Allen’s algebra, grouping its thirteen relations into before, during, and after classes while retaining all relations in the data.This interval-based representation supports compositions whose actions occupy extended temporal spans.
- Tasks: The dataset contains three progressively complex classification tasks: atomic action recognition, compositional action recognition, and snitch localization.The benchmark uses 5,500 videos per task setup, random 70:30 train-test splits, and parallel camera-motion data.
- Tasks: Snitch localization predicts the snitch’s final cell on a quantized 6 × 6 grid, requiring reasoning through occlusions and recursive containments.The task reports top-1, top-5, and mean L1 distance, with the latter reflecting grid-cell proximity.
4 EXPERIMENTS
Experiments evaluate CATER with spatiotemporal models, LSTM aggregation, and a tracking baseline across atomic actions, compositional actions, and snitch localization. Models struggle with long-term temporal structure, while diagnostics show sensitivity to camera motion, snitch motion and containment, and grid resolution.
- Task 1: Atomic action recognition: Adding more frames improves atomic action recognition, while R3D performs strongly with a static camera but substantially worse with a moving camera.
- CATER evaluates 14-way atomic action recognition, 301-way compositional action recognition, and 36-way snitch localization using different methods.
- Task 2: Compositional action recognition: Compositional action recognition is harder for existing models; non-local blocks and LSTM aggregation help, while a moving camera makes the task harder.
- Task 3: Snitch localization: Snitch localization benefits from longer clips and higher sampling rates, while TSN and flow perform worse than R3D on this task.
- Task 3: Snitch localization: The tracker becomes a stronger baseline as localization becomes finer-grained, whereas overall performance improves with a coarser grid.
- Localization diagnostics: Localization performance drops when the snitch keeps moving until the end or is contained in the final frame, but remains relatively stable as scene object count changes.
5 CONCLUSION
CATER exposes weaknesses in current video models on hard spatiotemporal tasks, especially long-term snitch localization. Its synthetic, controllable design supports diagnostics while complementing real-world benchmarks.
- Most leading models struggle on CATER, particularly with snitch localization requiring long-term reasoning.
- Average-pooled clip predictions and short temporal cues such as optical flow perform poorly on CATER.
- CATER should complement rather than replace real-world video benchmarks because it is synthetic and not a complete solution.
- The fully annotated dataset also supports richer parsing tasks beyond the classification tasks studied here.
A IMPLEMENTATION DETAILS FOR BASELINES
The baseline experiments use established ResNet-3D, non-local, and temporal segment network implementations under CATER-scaled training settings.
- Experiments use ResNet-3D, non-local blocks, and temporal segment networks for all baselines.
B TRAIN/VAL DISTRIBUTIONS
The appendix reports class distributions for the dataset’s defined tasks using training and validation data.
- Figure 6 shows the data distribution over classes for each defined task.
C VIDEO VISUALIZATION
The visualizations examine dataset examples, task difficulty, tracking behavior, model attention, and task distributions. They emphasize how occlusion, containment, and late snitch motion affect localization.
- Video samples: The supplementary video includes sample dataset videos with and without camera motion.
- Difficulty and tracking: The hardest validation videos involve sudden snitch motion near the end, whereas earlier stopping provides more evidence for the final location.
- Difficulty and tracking: Snitch tracking works in simple scenarios but fails under substantial occlusion or complex containment operations.
- Model attention: The model’s bottom-up attention concentrates on the snitch toward the end of clips, reflecting its importance for localization.
- Task distributions: The dataset distributions cover atomic actions, 588 spatiotemporal action compositions, and snitch localization.