PaperlayerView exact revision on arXiv

Source-linked AI summary

PerceptionComp: A Video Benchmark for Complex Perception-Centric Reasoning

Shaoxuan Li, Zhixuan Zhao, Hanze Deng, Zirun Ma, Shulin Tian, Zuyan Liu, Yushi Hu, Haoning Wu, Yuhao Dong, Benlin Liu, Ziwei Liu, Ranjay Krishna

arXiv:2603.26653v1cs.CVcs.AIcs.CLcs.LG

TL;DR

PerceptionComp addresses the lack of benchmarks for complex, long-horizon video reasoning that requires repeated perception and composition of temporally separated evidence. It introduces a manually annotated benchmark and finds that humans need substantially more time while current MLLMs achieve limited accuracy, indicating persistent challenges in perception-centric reasoning.

  • Problem

    Existing video reasoning benchmarks do not fully capture complex, long-horizon questions requiring repeated perception, temporal evidence gathering, and compositional constraints across multiple perceptual skills.

  • Method

    PerceptionComp is a fully manually annotated benchmark whose questions compose perceptual subconditions under conjunctive and sequential logic, with verification ensuring the full set is necessary for a unique answer.

  • Results

    Current MLLMs perform poorly on PerceptionComp: Gemini-3-Flash reaches 45.96% accuracy in the five-choice setting, while open-source models remain below 40%.

  • Takeaways & Limitations

    Perception-centric long-horizon video reasoning remains a major bottleneck, involving fine-grained evidence extraction and integration across multiple steps beyond generic model capacity.

  • Takeaways & Limitations

    The benchmark covers daily-life recordings but excludes high-stakes domains such as medical and surveillance settings, and it is not intended for identity recognition or sensitive-attribute inference.

Abstract

from arXiv · show

We introduce PerceptionComp, a manually annotated benchmark for complex, long-horizon, perception-centric video reasoning. PerceptionComp is designed so that no single moment is sufficient: answering each question requires multiple temporally separated pieces of visual evidence and compositional constraints under conjunctive and sequential logic, spanning perceptual subtasks such as objects, attributes, relations, locations, actions, and events, and requiring skills including semantic recognition, visual correspondence, temporal reasoning, and spatial reasoning. The benchmark contains 1,114 highly complex questions on 279 videos from diverse domains including city walk tours, indoor villa tours, video games, and extreme outdoor sports, with 100% manual annotation. Human studies show that PerceptionComp requires substantial test-time thinking and repeated perception steps: participants take much longer than on prior benchmarks, and accuracy drops to near chance (18.97%) when rewatching is disallowed. State-of-the-art MLLMs also perform substantially worse on PerceptionComp than on existing benchmarks: the best model in our evaluation, Gemini-3-Flash, reaches only 45.96% accuracy in the five-choice setting, while open-source models remain below 40%. These results suggest that perception-centric long-horizon video reasoning remains a major bottleneck, and we hope PerceptionComp will help drive progress in perceptual reasoning.

1 Introduction

PerceptionComp targets a gap in video benchmarks by requiring long-horizon, perception-centric reasoning that repeatedly gathers and composes temporally separated visual evidence. Human and MLLM evaluations show that this capability remains difficult.

  • Motivation and benchmark gap: Existing benchmarks either allow relatively easy single-view perception or derive difficulty mainly from non-perceptual reasoning, leaving repeated visual information gathering insufficiently measured.The paper identifies no prior benchmark that is simultaneously long-horizon, perception-centric, and demanding repeated visual evidence gathering.
  • Motivation and benchmark gap: PerceptionComp requires multiple temporally separated visual clues, compositional constraints, and perceptual skills including semantic, correspondence, temporal, and spatial reasoning.Questions use conjunctive and sequential logic, so no single video moment is sufficient.
  • Benchmark construction: 1,114 questions on 279 videos are manually annotated across diverse domains, including city walks, indoor villa tours, video games, and extreme outdoor sports.The benchmark uses a five-choice format with plausible, closely confusable options designed to require video evidence.
  • Human evaluation: Human participants take more than 2× longer than on VideoMMMU, more than 10× longer than on VideoMME, and more than 5× longer than on Video-Holmes and LongVideoBench.Most PerceptionComp videos are not longer than 10 minutes, so context length is not the only dimension of video thinking.
  • Model evaluation: 45.96% accuracy is achieved by Gemini-3-Flash in the five-choice setting, while open-source MLLMs remain below 40%, despite humans reaching 100% with unrestricted rewatching.Thinking models and larger test-time budgets help, but performance remains far from human-level.

2 Related Work

Prior video benchmarks broaden coverage or isolate individual reasoning skills, but typically require limited cross-moment evidence integration. Other multimodal reasoning benchmarks emphasize structured inference, symbolic tasks, or knowledge-heavy questions rather than complex perception-centric video reasoning.

  • General video understanding benchmarks: General video benchmarks commonly focus on local details, global summaries, broad task coverage, or isolated skills such as temporal reasoning and object interaction.Video-MME, ALLVB, MVBench, NExT-QA, and the Perception Test represent these directions.
  • General video understanding benchmarks: These video benchmarks are typically solvable with limited cross-moment evidence integration and remain comparatively easy as probes of long-horizon, compositional video thinking.Their relative ease limits their ability to assess repeated visual information gathering.
  • Complex multimodal reasoning benchmarks: Complex multimodal reasoning benchmarks evaluate mathematical, scientific, logical, or academic-style inference, often using visual inputs mainly as carriers of symbolic structure or domain knowledge.These tasks differ from perception-centric reasoning grounded in extended real-world video evidence.
  • Multimodal reasoning models: Reasoning-oriented LLMs motivate step-by-step inference and test-time scaling, while MLLMs increasingly accept visual inputs directly across proprietary and open-source model families.PerceptionComp applies this reasoning context to long-video perception rather than treating language-side inference as sufficient.

3 PerceptionComp

PerceptionComp is built from complex real-world videos and manually verified compositional questions that require coordinated perceptual skills. Its conjunctive and sequential designs enforce multi-step evidence integration across objects, attributes, relations, locations, actions, and events.

  • Video selection: PerceptionComp selects structurally complex videos with many objects, substantial motion, and frequent transitions, using SAM2 instance counts and optical-flow magnitude as complexity proxies.The videos span diverse categories and are difficult to summarize with a single caption.
  • Subconditions and perceptual skills: Questions combine subconditions targeting semantic, spatial, temporal, and correspondence skills, with some also requiring visual knowledge or near-future world modeling.These skills cover recognition, layout and occlusion, event ordering, cross-view identity matching, commonsense, and simple prediction.
  • Compositional question design: Conjunctive questions require every subcondition to constrain the same target, and no proper subset may uniquely determine the answer.This design prevents models from ignoring part of the query.
  • Compositional question design: Sequential questions resolve subconditions in order, carrying intermediate entities or states across later events and making early errors propagate through multi-hop perceptual reasoning.A question may identify an object, constrain later behavior, and then ask about a subsequent relation involving that object.
  • Answer space: Each question ends in a perceptual answer about an object, attribute, relationship, location, action, or event, presented as a five-way multiple-choice task with same-category distractors.Short, closely matched options reduce linguistic cues and discourage reliance on language priors.
  • Difficulty and annotation: Higher difficulty levels typically contain more subconditions, while expert annotation considers both subcondition count and intrinsic subcondition difficulty.The dataset contains 279 high-complexity videos and 1,114 questions, all manually annotated and verified for unique answers and necessary conditions.

4 Experiments

Experiments show that PerceptionComp remains difficult for current MLLMs, while performance improves when models receive more perceptual evidence or reasoning budget. Error analyses indicate that fine-grained perception, spatial constraints, and multi-step evidence integration remain major bottlenecks.

  • 4.2 Overall Results: Most models score below 40% on PerceptionComp, while Gemini-3-Flash achieves 45.96% accuracy.Strong open-source models remain lower, including Qwen3-VL-8B at 34.80% and Qwen3-VL-235B at 34.02%.
  • 4.2 Overall Results: Scaling model size does not consistently improve performance, suggesting that generic capacity is not the primary bottleneck.The benchmark instead stresses fine-grained evidence extraction under clutter and temporal discontinuities, followed by multi-step integration.
  • 4.2 Overall Results: Stronger test-time reasoning can help, but its benefits are not uniform across models and reasoning variants.GPT-o3 surpasses GPT-4o by 11.04%, while some Qwen3-VL thinking variants underperform their instruction-tuned counterparts.
  • 4.2 Overall Results: Accuracy drops substantially on Level 3 questions as compositional complexity increases.These questions require maintaining intermediate hypotheses while repeatedly gathering evidence across the video.
  • 4.3.1 Effect of Input Frames: 43.54% is GPT-o3 accuracy with 50 frames, up from 34.0% with 16 frames; Qwen3-VL-8B gains 7.8 points from 16 to 64 frames.Both models improve monotonically as the number of uniformly sampled input frames increases.
  • 4.3.2 Effect of Thinking Budget: Larger thinking-token budgets consistently improve Gemini-2.5-Flash accuracy.More tokens help maintain intermediate hypotheses, avoid premature commitment, and follow sequential subconditions.
  • 4.3 Analysis Experiments: PerceptionComp is sensitive to both perception and reasoning budgets, distinguishing visual recognition from sustained evidence-grounded multimodal reasoning.The benchmark therefore measures whether models can combine denser visual sampling with longer reasoning over temporally separated evidence.
  • 4.4 Case Study and Error Patterns: Qualitative failures often begin with incorrect fine-grained attributes or relations and then cascade through otherwise coherent reasoning chains.Gemini-3 failures peak at Step 3, and expert review attributes 60% of mid-stage failures to violated spatial subconditions.

5 Conclusion

PerceptionComp is a manually annotated benchmark for long-horizon video reasoning that requires repeated evidence gathering across time. Human and model results show that this capability remains difficult, while additional reasoning or perceptual compute helps but leaves a substantial gap.

  • 5 Conclusion: PerceptionComp contains 1,114 five-choice questions over 279 high-complexity videos and requires repeated evidence gathering across temporally separated segments.Its questions are designed so that no single moment is sufficient.
  • 5 Conclusion: Unrestricted rewatching enables 100% human accuracy, whereas state-of-the-art MLLMs reach at most 45.96% accuracy.The benchmark exposes a large gap between careful human performance and current model performance.
  • 5 Conclusion: Increasing test-time reasoning or perceptual compute helps models but leaves a large performance gap.The analysis varies reasoning tokens and input frames as distinct resources.

A Implementation Details

The evaluation process isolates each model’s definitive answer and uses an automated judge for models that produce complex reasoning.

  • A Implementation Details: Gemini-2.5-Flash serves as an automated judge for models employing complex reasoning strategies.The methodology programmatically removes intermediate reasoning and verbose output before verifying the final answer.

B More Analysis Experiments

A controlled human study examines how iterative access to video information affects PerceptionComp performance.

  • B More Analysis Experiments: Unconstrained human annotators achieve 85.10% overall accuracy in the controlled study.The study compares this condition with a stricter condition designed to isolate iterative perception and reasoning.

C Visualization Results

The visualizations show both disciplined multi-step reasoning and failures caused by abandoning the prescribed logic chain. Successful reasoning combines landmark grounding, temporal alignment, and fine-grained recognition, whereas failures involve spurious patterns and weak dependency tracking.

  • Successful reasoning: The successful case anchors on the glass-cube Apple Store, tracks yellow food trucks, and uses the passing yellow SUV to identify the bicyclist’s yellow-green vest.This chain demonstrates landmark grounding, spatio-temporal alignment, and detailed attribute recognition.
  • Failure cases: Models struggle when they abandon the prescribed reasoning path and replace required visual dependencies with invented interpretations or contextual heuristics.Observed failures include misclassifying benches, hallucinating alternative meanings, and targeting the wrong person despite observing an unrelated detail correctly.
  • Failure cases: Protagonist bias causes Qwen3-VL to prioritize the grey-clothed runner and behavioral consistency over fine-grained tracking of all four runners across timestamps.The model bypasses visual verification and reaches an incorrect elimination-based conclusion.
  • Implications: The failure cases indicate that long-horizon success depends on variable retention, suppression of contextual biases, and adherence to the prescribed logic chain.

D Gemini-3-Pro & Gemini-3-Flash

The comparison examines why Gemini-3-Flash outperforms Gemini-3-Pro on multi-step visual reasoning despite the latter’s greater reasoning capacity. The analysis attributes the inversion to sharper logical focus and fewer hallucinations from a streamlined inference process.

  • Performance comparison: Gemini-3-Flash achieves higher overall accuracy than Gemini-3-Pro on the multi-step visual reasoning benchmark.
  • Logical focus: Gemini-3-Pro often misallocates its longer reasoning chains to irrelevant fine-grained details instead of advancing the core logical sequence.Flash instead anchors on key logical clues and maintains sharper focus on long sequence data.
  • Hallucination control: Gemini-3-Pro’s over-analysis of fragmented spatial and temporal information can produce logical hallucinations and spatial confusion.The paper gives unnecessary absolute coordinate systems as an example.
  • Streamlining effect: The authors term Flash’s advantage the “streamlining effect,” arguing that streamlined inference can be more robust than unconstrained, overly complex reasoning in dynamic visual contexts.

E Analysis on Incorrect Step and Error Type

Error analysis shows that Gemini-3-Pro fails more often at later reasoning steps, with spatial understanding representing the largest categorized cause of error. These patterns indicate increasing difficulty as reasoning chains lengthen and persistent difficulty interpreting complex spatial relationships.

  • Incorrect reasoning steps: Errors occur most frequently in the second and third reasoning steps, at noticeably higher frequencies than in the first step.The distribution suggests that difficulty increases as the reasoning chain becomes longer.
  • Error types: The analysis categorizes failures into spatial understanding, static feature perception, incomplete reasoning, counting errors, and dynamic feature perception.Incomplete reasoning includes omitted conditions or logical inconsistencies.
  • Illustrative failure: The parkour failure illustrates how substituting narrative assumptions for precise tracking can produce incorrect conclusions across temporally separated observations.Qwen3-VL fixates on the grey-clothed runner instead of verifying all four runners’ flip directions.
  • Error types: Spatial understanding accounts for the largest proportion of failures among the analyzed error categories.This indicates difficulty accurately interpreting spatial relationships in complex visual scenes.

F Limitations and Ethical Considerations

PerceptionComp is scoped to complex perception-centric reasoning in long videos and does not represent every video reasoning setting. Its coverage and release are bounded by domain exclusions, source-use permissions, and restrictions on sensitive attributes and interpretation.

  • Scope: PerceptionComp covers daily-life recordings but excludes high-stakes domains such as medical and surveillance settings.
  • Data governance: The benchmark uses only videos whose usage is compatible with academic benchmarking and does not release content beyond original source terms.
  • Ethical boundaries: The annotations exclude personally sensitive attributes, and the benchmark is not intended for identity recognition, surveillance, or sensitive-attribute inference.
  • Evaluation considerations: Because performance can hinge on multi-step logical adherence and spatial reasoning, the authors caution against relying solely on absolute accuracy.They recommend interpreting comparative performance trends while considering possible biases and heuristics.

F.1 Annotation protocol and quality control

PerceptionComp uses trained annotators and two-stage review to construct and validate compositional questions. Independent checking found 89.0% agreement, indicating that most sampled questions have clear, unambiguous solutions.

  • Annotators receive technical-background training, study 20 example videos and questions, and pass calibration before contributing.
  • In stage one, an annotator watches the video, proposes a compositional question with three subconditions, and specifies one correct answer.
  • A second annotator re-watches each question and checks answer correctness, solution uniqueness, and necessity of every subcondition; failed items are revised or discarded.
  • 89.0% agreement between a third annotator and the original answer key across 100 sampled questions indicates that most questions admit clear, unambiguous solutions.
Loading 2603.26653v1…