Source-linked AI summary
The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks
Shuo Wen, Beixi Du, Edwin Meriaux, Junming Shi, Chloe Si, David Meger, Doina Precup, Gregory Dudek
TL;DR
Professional magic exploits human expectation and misdirection, making it a test of visual reasoning across extended narratives. The Prestige benchmarks that reasoning with open-ended questions about magic performances. Across 14 VLMs, the best model remains substantially below human performance, while multiple-choice formats raise measured accuracy over open-ended answers.
Problem
Magic performances challenge models to track visual cues, attention, and expectations over long sequences, motivating a benchmark of cognitive visual reasoning.
Source: §1 · p. 2Method
The Prestige evaluates VLMs on 2,177 open-ended questions about professional magic, spanning short-term, multi-step, long-range, and adversarial visual reasoning.
Source: §1 · p. 2Results
Across 14 VLMs, Gemini3.1-Pro scores 47.08%, 45.71 percentage points below humans; on 200 questions, four-choice accuracy exceeds open-ended accuracy by 34.0 points.
Source: §6 · p. 9Takeaways & Limitations
The multiple-choice accuracy increase supports using open-ended questions to assess whether models can generate answers independently.
Source: §1 · p. 2, §6 · p. 9The benchmark releases YouTube video identifiers rather than full videos, and sharing archived copies depends on creator permission and applicable licenses.
Source: §6 · p. 10
Abstract
from arXiv · showhide
We introduce The Prestige, a video understanding benchmark that probes cognitive visual reasoning through professional magic performances, where deliberate misdirection challenges models to track events and explain unexpected outcomes. The benchmark comprises open-ended questions ranging from temporal and causal understanding to robustness against adversarial questions. Evaluation of leading open-source and proprietary VLMs reveals a substantial gap between even the strongest model and human performance. We also find systematic discrepancies between human and automated grading and identify recurring model failure modes. Ablations further show that models can select correct multiple-choice answers even when they fail to generate correct explanations independently, motivating further investigations into question formats. Together, these findings highlight limitations in both VLM reasoning and its evaluation, underscoring the need for more reliable video understanding models and more rigorous benchmarks.
1 Introduction
The paper argues that professional magic tests visual reasoning by exploiting expectations, misdirection, and long-range causal tracking. The Prestige evaluates these abilities with open-ended questions and finds a substantial gap between leading VLMs and humans, while showing that multiple-choice formats inflate performance.
1 Introduction
- Magic exploits commonsense expectations and multi-step reasoning, making it a setting for testing how systems track visual cues and respond to counter-intuitive outcomes.The paper calls the compositional process of tracking long-context cues, focusing attention, and building predictive mental models “cognitive flow.”Source: §1 · p. 2
- Professional magic supports visual-only evaluation, long-horizon memory and causal tracking, and reduced reliance on memorized trick mechanics.Performances can foreshadow elements that are revealed later, while advanced sleight mechanics are closely guarded.Source: §1 · p. 2
- The benchmark targets a gap in video reasoning evaluation: multiple-choice formats can encourage shortcuts and offer limited evidence of independent reasoning and explanation.The paper positions this concern against benchmarks seeking to evaluate distant temporal dependencies, multi-step reasoning, and causal visual clues.Source: §1 · p. 2
- The Prestige introduces 2,177 open-ended questions across six categories, testing VLM visual reasoning from short-term perception to complex cognitive flow.The questions are grounded in visually sufficient professional magic performances.Source: §1 · p. 2
- Evaluations show a substantial performance gap between state-of-the-art VLMs and human baselines.The paper also identifies limited cognitive flow, weak long-context reference, and unreliable visual identification as recurring bottlenecks.Source: §1 · p. 2
- An ablation finds that multiple-choice formats artificially inflate model performance, motivating strictly open-ended questions for more rigorous evaluation.The authors argue open-ended formats better assess independently generated answers.Source: §1 · p. 2
2 Related Work
The related work describes advances in video-language models and increasingly long-context video benchmarks. The Prestige distinguishes its focus on cognitive flow in staged magic routines from prior evaluations that do not centrally assess this kind of reasoning.
2.2 Benchmarks for video understanding
- Table 1 compares video-understanding benchmarks by question format and whether cognitive flow is centrally evaluated.The caption defines MC as multiple choice, QA as open question-answering, and △ as partial coverage.Source: Table 1 · p. 3
2.1 VLMs for video understanding
- Recent VLMs have advanced multimodal perception, reasoning, and long-video comprehension, with some work emphasizing efficient long-context modeling and reasoning over extended streams.The section also notes advances in proprietary systems such as GPT-5 and Gemini3.1-Pro.Source: §2.1 · pp. 2–3
2.2 Benchmarks for video understanding
- Video benchmarks cover temporal understanding, long-range evidence, causal reasoning, and narrative reasoning, but the cited settings do not examine deeper cognitive flow.The section identifies FunQA as the closest prior benchmark, while noting that it mixes magic with other counterintuitive videos and treats magic mainly as a local anomaly.Source: §2.2 · p. 3
3 Methodology
The methodology selects professional, visually comprehensible magic performances and builds peer-reviewed open-ended questions to test varied visual reasoning abilities. The resulting benchmark contains 180 videos and 2,177 questions spanning six categories.
3.1 Video Collection
- Videos are selected for professional quality, ethical standards, high resolution, visual-only comprehensibility, causal coherence, and contextual density.The performances must not depend on audio or text and must not be captioned.Source: §3.1 · p. 3
3.2 Question and Answer generation
- Question-answer pairs are constructed to evaluate distinct visual, temporal, and commonsense reasoning abilities in multi-step and long-dependency settings.Figure 2 illustrates the categories with sample questions, human answers, and relevant video frames; evaluation samples frames uniformly at 1 FPS, up to four minutes.Source: §3.2 · p. 3, Figure 2 · p. 4
- The six categories test current-frame reasoning, context-dependent memory, event summarization, explanations of surprising tricks, and rejection of false implications or statements.Adversarial answers must challenge false premises and correct them using visual evidence; temporal summaries capture core actions rather than low-level details.Source: §3.2 · pp. 4–5
3.3 Dataset evaluation
- All generated questions undergo peer review against the Question-Answer Quality Standard.The paper refers to an appendix for the standard and review process.Source: §3.3 · p. 5
3.4 Statistics of the dataset
- The final benchmark contains 180 videos and 2,177 open-ended questions from unique routines by over 150 magicians.Contextual Reasoning and Paradoxical Reasoning account for 36.2% and 34.4% of questions, respectively; the questions also span varied props and effects.Source: §3.4 · p. 5
4 Experiments
The experiments compare VLMs with human viewers, examine differences between human and machine grading, and analyze recurring errors. The strongest model remains far below human performance, while sampled errors cluster around cognitive-flow reasoning, long-context reference, and visual identification.
4 Experiments
- The evaluation compares long-video and multi-frame VLMs using frames sampled at 1 FPS from up to four minutes before each question’s timestamp.Each question was answered by an independent reviewer under the models’ evaluation protocol to establish the human baseline.Source: §4 · p. 5
4.1 Benchmark Results
- Gemini3.1-Pro scores 47.08% overall, 45.71 percentage points below humans at 92.79%; no model leads every question category.Gemini3.1-Pro leads PR, AQ.FI, and AQ.FS; Gemini3.1-Flash-Lite leads IR and TS, while Qwen3.6-Plus leads CR. Human accuracy is lowest on AQ.FS (83.70%) and TS (90.18%).Source: §4.1 · pp. 5–6, Table 2 · p. 6
4.2 Comparison of Human and Machine Grading
- Human grading exceeds machine grading by a mean of 4.56 percentage points (SD 1.23), but the difference varies by model and question type.TS has the largest mean difference (7.40 points) and CR the smallest (3.39 points); negative differences in some entries show machine grading is not uniformly stricter. These aggregate differences do not establish agreement on individual answers.Source: §4.2 · p. 7, Table 3 · p. 7
4.3 Error Analysis
- Among 100 randomly sampled errors per model, cognitive-flow, long-context-reference, and visual-identification errors account for 67%, 66%, and 65% of errors, respectively.The comparisons are descriptive; small differences may reflect sampling variation or borderline annotations, and question-type associations may affect observed differences. Together with Table 2, the analysis highlights difficulty rejecting false premises and following a routine’s cognitive flow.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
5 Ablations on the effects of MC Questions
Multiple-choice formats sharply raise measured accuracy over open-ended answers, including on questions every evaluated model initially failed. The comparisons show that selecting a candidate answer can conceal difficulty generating an adequate answer independently.
Experimental setup
- The experiments compare open QA with 8-way and 4-way multiple choice on random and jointly failed question samples across three VLMs.Each MCQ includes the correct answer and distractors generated by an LLM; option order is randomized.Source: §5 · p. 8
Randomly sampled questions
- On 200 random questions, mean accuracy rose from 37.5% open-ended to 67.2% with 8-way and 71.5% with 4-way multiple choice.The 4-way gain was 34.0 percentage points; Figure 7(b) reports the comparison.Source: §5 · p. 8, Figure 7 · p. 9
Failure-selected questions
- On 50 questions all three models failed open-ended, mean 4-way accuracy gained 73.3 percentage points.Figure 7(a) presents these failure-selected questions; open-ended accuracy was 0% by construction.Source: §5 · p. 8, Figure 7 · p. 9
Implications
- Providing candidate answers substantially increases measured accuracy and can conceal failures to generate an adequate answer independently.The authors call for further study of question formats for qualitative reasoning tasks.Source: §5 · p. 9
6 Discussion & Conclusion
The Prestige benchmarks cognitive visual reasoning with open-ended questions about professional magic, where models must interpret long-range evidence and unexpected outcomes. Results show a substantial human–model gap, sensitivity to grading protocol, and large accuracy increases when candidate answers are supplied.
6 Discussion & Conclusion
- Gemini3.1-Pro scored 47.08%, 45.71 percentage points below the 92.79% human baseline across the benchmark’s 2,177 open-ended questions.The benchmark evaluates 14 leading proprietary and open-source VLMs; sampled errors include failures in cognitive flow, context referral, and visual identification.Source: §6 · p. 9
- Machine grading scored all 14 models lower overall than human grading, with a mean difference of 4.56 percentage points and standard deviation 1.23.Differences varied by question category, including combinations where machine scores exceeded human scores.Source: §6 · p. 9
A Magic as a testbed for cognitive visual reasoning
- A magic reveal tests whether a model links visible setup, commonsense expectations, temporal state changes, and the observed discrepancy.Misdirection makes relevant evidence compete with noticeable but irrelevant actions, while the dataset excludes audio narration and captions.Source: A Magic as a testbed for cognitive… · p. 13
B Why moving away from MC
- Open-ended answers expose whether models can give a specific causal account, whereas multiple-choice options can enable token, position, or elimination shortcuts.Such shortcuts may obscure whether the model tracked the video or fabricated its reasoning.Source: B Why moving away from MC · pp. 13–14
C Dataset comments
The benchmark prioritizes reliable, visually grounded questions about long-horizon cognitive reasoning over uniform coverage of video-understanding skills. Its quality standards constrain both source videos and question-answer items to support visual sufficiency and answerability.
C Dataset comments
- The dataset is smaller than general-purpose video-QA benchmarks and deliberately weights coverage toward contextual and paradoxical reasoning rather than all video-understanding skills.The authors prioritize video quality, question depth, and annotation reliability over dataset size.Source: C Dataset comments · p. 14
- Models were evaluated on frames sampled at 1 FPS because of model and API constraints.The authors encourage future evaluation at native frame rates to preserve fine-grained temporal information.Source: C Dataset comments · p. 14
D Quality Standards
- Source videos must be professional, visually self-sufficient, longer than 90 seconds, and sufficiently clear and causally coherent to include multiple surprises.The standards exclude amateur recordings, narration-dependent reveals, hardcoded captions, glitches, and videos with inadequate visual fidelity or contextual density.Source: D Quality Standards · p. 14
- Question review removes items answerable from text alone, visually ungrounded, dependent on outside knowledge, mistimed, incorrect, or mismatched to their reasoning type.Reviewers independently answer questions from the video, resolve disagreements through rewatching and escalation, and check phrasing and stylistic consistency.Source: D Quality Standards · p. 15
E Dataset Generation Pipeline
The dataset-generation pipeline applies video and question-answer quality standards from sourcing through final record creation. Candidate performances are manually reviewed end-to-end, and any failure to meet a video criterion leads to exclusion.
E Dataset Generation Pipeline
- The pipeline combines video sourcing, two-pass QA annotation, peer review, and a final item schema under shared video and QA quality standards.These stages are intended to keep each item visually grounded, unambiguously answerable, and aligned with its target reasoning skill.Source: E Dataset Generation Pipeline · p. 15
E.1 Video Sourcing Protocol
- Candidates came from publicly available professional magic performances and were retained only after manual end-to-end review against every video-quality criterion.No automated pre-filtering was used; common rejection reasons included audio-dependent reveals, aggressive editing, and short single-effect clips.Source: §E.1 · p. 15
E.2 Question Generation Protocol
Question generation uses a two-pass annotation process: annotators first log state-changing moments, then write timestamped question-answer pairs. Peer review checks quality, with unresolved disagreements escalated to a third annotator.
E.2 Question Generation Protocol
- Annotators first scan each video without audio or pauses, logging moments when object states change to create a dense set of candidate question points.They may restart the video if an important event was missed.Source: §E.2 · pp. 15–16
- After viewing the full video, annotators choose question types and draft question-answer pairs anchored to explicit timestamps.Each item also receives keywords identifying its dominant prop or technique.Source: §E.2 · p. 16
E.3 Peer Review Protocol
- A peer reviews the questions for quality, and unresolved disagreements go to a third annotator whose decision is final and logged.Records enter the benchmark only after passing every checklist item.Source: §E.3 · p. 16
E.4 Final Record Schema
The dataset stores question-answer records grouped by source video, alongside metadata. Each record carries an identifier, timestamp, question type, keywords, question, and answer.
E.4 Final Record Schema
- Each source video maps to a Questions list and metadata block, colocating records from the same performance for joint retrieval.The dataset is structured as one JSON file keyed by source video identifier.Source: §E.4 · p. 16
- A record is a tuple containing a UID, timestamp, question type, keywords, question, and answer.The example record shows a timestamped Paradoxical Reasoning question about why people are surprised.Source: §E.4 · pp. 16–17
- The metadata start field marks when the performance begins, letting evaluation skip introductions; this field is not provided to the model.The video_id is the source clip’s YouTube identifier, while the timestamp anchors a question to a frame range and defines the evaluation context window.Source: §E.4 · pp. 16–17
F Question Types
The question taxonomy separates frame-level identification, cross-frame tracking, surprise explanation, premise checking, and temporal compression. The accompanying grading protocols use binary judgments, with answers assessed against the ground truth and question-specific requirements.
F.1 Instant Reasoning (IR)
- Instant Reasoning (IR) tests whether the model identifies the query-relevant state at one timestamp, providing a foundation for downstream reasoning.It also surfaces temporal slippage when an answer relies on a nearby but incorrect frame.Source: §F.1 · p. 17
F.2 Contextual Reasoning (CR)
- Contextual Reasoning (CR) asks for a specific answer that depends on earlier video evidence unavailable in the queried frame alone.It tests retrieval of relevant cross-frame memory, such as an object’s earlier identity or state through occlusion.Source: §F.2 · pp. 17–18
F.3 Paradoxical Reasoning (PR)
- Contextual Reasoning (CR) and Paradoxical Reasoning (PR) both require cross-frame evidence, but PR additionally reconstructs the expected outcome and explains its conflict with the reveal.PR may require tracing an earlier action through misdirection to identify how the audience’s expectation formed.Source: §F.2 · p. 17, §F.3 · p. 18
F.4 False Implication Question (AQ.FI)
- False Implication questions embed a false premise indirectly, while False Statement questions assert one explicitly; both test whether models verify premises against visual evidence.False Statement answers must reject or correct the premise and provide the replacement fact.Source: §F.4 · p. 18, §F.5 · p. 18
F.6 Temporal Summarization (TS)
- Temporal Summarization (TS) tests whether models compress a specified interval while preserving essential events, rather than enumerating frames or overgeneralizing.Its characteristic errors include granularity mismatch and omission of a critical action.Source: §F.6 · p. 19
G.1 Grading Protocol
- Human evaluators provide the primary scores, while machine evaluation uses three API-based VLM judges whose binary grades are combined by majority vote.The protocols report human and machine scores separately for comparison.Source: §G.1 · pp. 19–20, G Grading Pipeline · p. 19
- The grading rubric requires answers to match the ground truth’s key facts and specificity, while applying question-type-specific criteria and rejecting unsupported or contradictory claims.Adversarial answers must reject false premises; PR answers must explain the surprise with specific details.Source: §G.1 · pp. 20–21
H Evaluation Prompt
- The evaluation prompt frames the model as an audience without prior magic knowledge and grades only the response in the answer tags.The reasoning component is retained for diagnosis but excluded from grading.Source: H Evaluation Prompt · p. 22
I Machine Grading Results
The section identifies human grading as the primary benchmark evaluation and presents a table comparing human and majority-vote machine scores across models and question categories.
I Machine Grading Results
- Human grading remains the primary benchmark evaluation, while Table 3 reports differences between human and majority-vote machine scores for all 14 models.Table 4 reports machine-evaluated model accuracy by question type across 14 models over the full dataset.Source: I Machine Grading Results · p. 23, Table 4 · p. 23