PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception Test: A Diagnostic Benchmark for Multimodal Video Models

Viorica Pătrăucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimczak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zisserman, João Carreira

arXiv:2305.13786v2cs.CVcs.AIcs.LG

TL;DR

Multimodal video benchmarks often cover restricted computational tasks, leaving broad perception and reasoning insufficiently diagnosed. The Perception Test addresses this gap with purposefully designed real-world videos, dense multimodal annotations, and transfer-focused evaluation; baseline models remain far from human performance on several skills and reasoning types.

  • Problem

    Existing benchmarks focus on restricted computational tasks, motivating broader evaluation of multimodal perception and reasoning across skills and modalities.

  • Method

    The Perception Test uses purposefully scripted and filmed real-world videos with six annotation types, evaluated in zero-shot, few-shot, or limited-fine-tuning settings.

  • Results

    Flamingo and SeViLA are far from human performance and cannot outperform the 8-shot dummy frequency baseline on the benchmark's diagnostic evaluations.

  • Takeaways & Limitations

    The benchmark provides broad diagnostics, open training and validation resources, per-task baselines, and a challenge server for held-out testing.

  • Takeaways & Limitations

    Broad coverage makes perfect balance across all skills, reasoning types, and modalities challenging, and future work aims to add more counterfactual tasks.

Abstract

from arXiv · show

We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GPT-4). Compared to existing benchmarks that focus on computational tasks (e.g. classification, detection or tracking), the Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and types of reasoning (descriptive, explanatory, predictive, counterfactual) across video, audio, and text modalities, to provide a comprehensive and efficient evaluation tool. The benchmark probes pre-trained models for their transfer capabilities, in a zero-shot / few-shot or limited finetuning regime. For these purposes, the Perception Test introduces 11.6k real-world videos, 23s average length, designed to show perceptually interesting situations, filmed by around 100 participants worldwide. The videos are densely annotated with six types of labels (multiple-choice and grounded video question-answers, object and point tracks, temporal action and sound segments), enabling both language and non-language evaluations. The fine-tuning and validation splits of the benchmark are publicly available (CC-BY license), in addition to a challenge server with a held-out test split. Human baseline results compared to state-of-the-art video QA models show a substantial gap in performance (91.4% vs 46.2%), suggesting that there is significant room for improvement in multimodal video understanding. Dataset, baseline code, and challenge server are available at https://github.com/deepmind/perception_test

1 Introduction

The Perception Test addresses the limitations of task-specific perception benchmarks with a multimodal diagnostic benchmark spanning skills, reasoning types, and annotation formats.

  • Motivation: Existing benchmarks typically target restricted computational tasks, limiting coverage across temporal, semantic, and low-level perception capabilities.Benchmarking is presented as important for ranking model designs and identifying improvement areas.
  • Benchmark proposal: The Perception Test evaluates Memory, Abstraction, Physics, and Semantics through descriptive, explanatory, predictive, and counterfactual reasoning across video, audio, and text.Its videos are purposefully designed, filmed, and annotated to expose perceptually interesting situations.
  • Benchmark contents: 11.6K real-world videos are densely annotated with object and point tracks, action and sound segments, and multiple-choice and grounded video-question answers.These annotations support both language and non-language evaluations.
  • Resources and evaluation: The benchmark provides open training and validation data, a held-out-test evaluation server, and per-task baselines with diagnostic reports across skills and reasoning types.The paper notes that no existing model handles all evaluation tasks.

2 Related work

The Perception Test builds on video benchmarks and diagnostic datasets but emphasizes broad skill coverage, dense annotations, and transfer evaluation rather than large-scale training.

  • Prior benchmarks: Prior real-world benchmarks draw on web videos, volunteer-filmed activities, or scripted crowd-sourced recordings, each providing different data-collection settings.The paper compares these sources in terms of coverage, modalities, annotations, and tasks.
  • Motivation: Public-video annotation is difficult for evaluation because diverse perceptually interesting situations are costly to find, filter, and annotate, and some situations are unavailable.The authors therefore design scripts and film participants from different places in the world.
  • Diagnostic benchmarks: Synthetic diagnostic datasets systematically test intuitive physics, while the Perception Test extends diagnostic coverage to real-world multimodal perception.The shared motivation is to evaluate abilities such as collisions, motion, and object permanence alongside broader skills.
  • Positioning: The Perception Test has broader skill coverage and higher annotation density than prior efforts, while remaining an evaluation benchmark rather than a large-scale training dataset.It uses limited fine-tuning or prompting data to assess model transfer capabilities.

3 Videos in the Perception Test

The dataset uses scripted, varied household scenarios filmed by a globally diverse participant pool to probe selected perception skills and reasoning types.

  • Script design: Scripts target Memory, Abstraction, Physics, and Semantics through descriptive, explanatory, predictive, and counterfactual reasoning.Skill selection reflects blind spots in existing benchmarks, weaknesses of current models, and aspects important for real-world scene understanding.
  • Script design: Each script describes a simple household situation, its actions, and camera placement so that a non-professional actor can perform it.The scenarios use ordinary household items or easily crafted substitutes.
  • Task difficulty: The benchmark links high-level concepts to low-level tasks and introduces distractor actions to make otherwise simple situations more challenging.Examples include testing memory through object tracking and physics through question answering.
  • Dataset diversity: 37 scripts with 2–5 variations each support diverse videos and reduce language bias by changing answers while keeping questions and options fixed.The correct answer depends on the specific script variation.
  • Video filming: Around 100 participants from different countries, ethnicities, and genders filmed the script variations to diversify participants and depicted scenes.Each variation was filmed by at least a dozen different participants.
  • Splits: The dataset contains 11609 videos with audio averaging 23 seconds, divided into training, validation, and held-out test splits.The training split supports fine-tuning or prompting, while the test split is accessed through an evaluation server.

4 Annotations in the Perception Test

Six annotation types connect dense spatial and temporal supervision with language-based reasoning, enabling complementary low-level and high-level evaluations.

  • Annotation types: The six annotation types are object tracks, point tracks, temporal action segments, temporal sound segments, multiple-choice videoQA, and grounded videoQA.Together they support spatial, temporal, language, and non-language evaluations.
  • Spatial annotations: Object tracks form the annotation root, linking nearly all other annotations to objects involved in interactions or nearby distractor objects.Bounding boxes are annotated throughout videos at 1 fps.
  • Spatial annotations: Point tracks complement boxes by representing surface motion and deformation for articulated, non-rigid, thin, or rotating objects.They address physical interactions that bounding boxes do not fully describe.
  • Temporal annotations: Action segments use templated labels and action-relevant object tracks, with boundaries defined by contact and changes in action type.This provides temporal grounding for actions.
  • Temporal annotations: Sound segments are temporally labeled from 16 audio categories and grounded to the object or objects producing each sound, including off-screen sources.Annotators use both the video and its audio to assign these labels.
  • Question answering: Video-level questions cover descriptive, explanatory, predictive, and counterfactual reasoning, while grounded questions return answer-relevant object tracks.The question-answer annotations connect high-level reasoning to low-level spatial grounding.

5 Computational tasks and baseline results in the Perception Test

The Perception Test evaluates six multimodal tasks with per-task baselines, revealing substantial difficulty across tracking, localisation, and video question-answering. Human accuracy on multiple-choice video QA is high, while current video-language models remain far behind and often fail to exceed a frequency baseline.

  • Task design: The benchmark combines object and point tracking, temporal action and sound localisation, and multiple-choice and grounded video question-answering.Video and audio are available for all tasks.
  • Tracking: 0.56 mean IoU for Stark versus 0.67 for SiamFC shows that the simpler Siamese tracker performs better in zero-shot evaluation.Tracking performance also drops when the camera or tracked objects move.
  • Tracking: 0.466 versus 0.401 is the reported TAP-Net performance comparison between the Perception Test and Kinetics.The authors mainly attribute the drop to the benchmark’s longer videos: 23 seconds versus 10 seconds.
  • Localisation: 15.56 mAP on the Perception Test versus 22.7 mAP on EPIC-Kitchens is reported for the fine-tuned ActionFormer action-localisation baseline.Rare actions and pretend actions are frequent confusion sources, and multimodal inputs do not significantly increase performance.
  • Localisation: The best sound-localisation performance uses both video and audio features as input.The same ActionFormer model was adapted for audio localisation.
  • Video QA: 91.4% overall accuracy was achieved by humans on the multiple-choice video-QA study.Participants answered 126 questions in total, with 10 answers per question.
  • Video QA: 0-shot and few-shot Flamingo and SeViLA remain far from human performance and cannot outperform the 8-shot frequency dummy baseline.On some Memory, Physics, and Abstraction skills, model performance falls below that dummy baseline and occasionally below random; SeViLA fine-tuning improves results to some extent.

6 Conclusion

The paper concludes that the Perception Test is a broad diagnostic benchmark for multimodal perception and reasoning using purposefully designed real-world videos. Its reported baselines show a large gap between current video-language models and human performance.

  • Conclusion: The benchmark probes memory, abstraction, physics, and semantic capabilities across visual, audio, and text modalities.It uses real-world videos designed to show interesting perceptual situations and requires descriptive, explanatory, predictive, and counterfactual reasoning.
  • Conclusion: Six annotation types support object and point tracking, action and sound localisation, and multiple-choice and grounded video question-answering.The videos and annotations are open-sourced for the training and validation splits, with an evaluation server for the held-out test split.
  • Conclusion: State-of-the-art zero-shot or few-shot video-language models do not outperform a dummy frequency-based baseline, whereas humans in the same setting are nearly perfect.The conclusion frames the benchmark as a setting where an ideal perception model would perform all tasks.

1 Perception Test at a glance

The presentation materials summarise the Perception Test’s videos, annotations, and computational tasks.

  • Overview: Figure 1 and the presentation video summarise the types of videos, annotations, and tasks available in the Perception Test.

2 More details about annotations in the Perception Test

The appendix details how object and point tracks are distributed across camera motion and related task contexts, alongside distributions of words, actions, sounds, and questions.

  • Object tracks: Object tracks are distributed across camera motion and whether they are involved in actions, sounds, or grounded video question-answering.
  • Point tracks: Point tracks are split by point and camera motion.
  • Additional distributions: The appendix includes distributions for popular object-name words, actions, sounds, and questions across skills.Figure A3 specifically shows the distribution of questions across skills.

3 Computational tasks

The Perception Test defines six computational tasks spanning object and point tracking, temporal action and sound localisation, and two forms of video question-answering. These tasks use annotations and metrics designed to evaluate both language and non-language perception.

  • Tracking: Object tracking follows each labelled object from an initial bounding box and evaluates average IoU over the complete track.For approximately 20% of objects entering the frame, a later representative query frame is selected.
  • Tracking: Point tracking traces ground-truth initial 2D points through the video and evaluates position, occlusion, and combined Jaccard accuracy.
  • Temporal localisation: Temporal action and sound localisation predict and classify potentially overlapping one-dimensional temporal segments using mean average precision based on temporal IoU.
  • Video question-answering: Multiple-choice video question-answering asks models to select the correct answer from three options, with 33% random chance, and is scored by top-1 accuracy.Some questions require the options to choose among generic statements, while certain videos must be trimmed to prevent answer leakage.
  • Video question-answering: Grounded video question-answering conditions object-track retrieval on a language question and evaluates the resulting tracks with HOTA metrics.

4 Baseline results

Baseline results show that model performance varies substantially by task and setting: specialised trackers can outperform simple baselines under motion, while video QA models remain well below human performance. Action localisation is particularly challenged by rare and pretend-action classes.

  • Point tracking: 0.328 average Jaccard for TAP-Net improves on 0.088 for the static baseline when points move in the moving-camera setup.TAP-Net also improves occlusion accuracy from 0.675 to 0.849, while outperforming the static baseline on position and Jaccard measures above 4 pixels.
  • Temporal localisation: ActionFormer struggles with rare actions and confuses pretend actions with their non-pretend counterparts; audio-only localisation performs very poorly.Adding multimodal video and audio features does not significantly increase performance over the other configurations.
  • Video question-answering: Flamingo and SeViLA cannot outperform the 8-shot dummy frequency baseline, with some skills falling below the random baseline.The below-random cases include conservation, collision, and counterfactual reasoning tasks.
  • Video question-answering: 91.4% human accuracy exceeds SeViLA’s 62.0 fine-tuned performance and 46.2 zero-shot performance on video question-answering.The benchmark also reports Flamingo evaluations, frequency-based dummy baselines, and per-skill results across memory, abstraction, physics, and semantics.

5 Dataset Splits Generation

The dataset splits are generated to separate script-participant groups while controlling video-attribute proportions. A linear program assigns these groups using binary variables and bounded deviations from target split fractions.

  • Split construction: Videos are split into train, validation, and held-out test sets in roughly 20%/50%/30% proportions.
  • Split construction: All videos sharing a unique script_id and participant_id combination remain in the same split.This keeps related script variations and camera configurations together.
  • Question coverage: The split constraints distribute videos associated with each multiple-choice and grounded QA question across splits in the specified proportions.
  • Linear-program formulation: The split-generation problem uses binary decision variables for 7,288 unique script-participant pairs and 249 video attributes.The feature-count matrix records how many videos in each pair have each attribute.
  • Linear-program formulation: Each attribute’s achieved split fraction is constrained to lie between (1 − λ) and (1 + λ) relative to its target fraction.Here λ = 0.25, and the constraints are implemented through a linear program solved with CVXPY and MOSEK.

6 Annotation collection

The Perception Test combines multimodal annotation procedures with checks for cross-modal consistency, participant diversity, and several documented scope boundaries. Its annotations support language and non-language evaluation while retaining controlled design choices for coverage and analysis.

  • Annotation procedures: Annotators created object and point tracks by marking objects or semantic points and correcting tracker-generated intermediate proposals.Object annotation included interacted, nearby distractor, and background objects; point tracks required at least three points across different object parts.
  • Annotation procedures: Annotators marked action and sound segment boundaries and linked each segment to the object tracks involved.Action boundaries were tied to contact moments, while sound annotations identified the objects producing the sound.
  • Annotation procedures: Multiple-choice videoQA used an open-ended collection phase followed by answer selection from options with one correct answer.Negatives were sampled from answers to other videos in the same script type and checked to avoid ambiguous distractors.
  • Quality checks: Strong correlations between action-sound object pairs provided a sanity check for consistency across modalities.Examples included pouring with fluid interaction sounds, clapping with human clapping sounds, and moving objects with rolling sounds.
  • Participant diversity: Participant filming covered gender, ethnicity, and country of residence, but did not fully cover factors such as age, disability, income, or education.The authors report balanced gender representation, broad ethnicity coverage, and filming in more than 13 countries, based on self-reported demographics.
  • Scope and limitations: The benchmark balances broad multimodal coverage against practical constraints, including imperfect task balance, missing modalities, medium scale, controlled tabletop scenes, and no active-perception evaluation.The authors identify missing touch, force, deformations, detailed 3D geometry, and active interaction, while using scripted variations to test correct actions and consequences.
Loading 2305.13786v2…