PaperlayerView exact revision on arXiv

Source-linked AI summary

Humanity's Sixth Sense: Benchmarking Intuitive Visual Reasoning in Multimodal Models

Xingang Guo, Jing Gu, Brian Jang, Renxiong Wang, Utkarsh Tyagi, Daniel Quigley, Steven Li, David Yan, Daniel Yue Zhang, Darvin Yi, Forrest Huang, HiJae Kim, Tianyi Zhang, Jared Lichtarge, Jihua Huang, Le Xue, Manan Tomar, Qiuyi Richard Zhang, Ruofei Yu, Seth Neel, Yaning Hu, Marcella Valentine, Xinzhe Jiang, Daniel Evans, Chenguang Wang, Dustin Tran, Tong Zhao, Yinfei Yang, Yunzhong He

arXiv:2610.08966v2cs.AIcs.MM

TL;DR

Existing visual benchmarks emphasize specialized academic problem-solving or low-level perception, leaving intuitive inference of implicit scene information largely untested. HSS benchmarks that capacity across image and video tasks, but frontier models remain far below humans. GPT-6-astra reaches a 53.6% pass rate versus 93.1% for human participants, even at maximum reasoning effort.

  • Problem

    Existing visual benchmarks emphasize specialized academic problem-solving, low-level perception, or fragmented probes, leaving intuitive inference of implicit visual information largely untested.

    Source: §1 · p. 2

  • Method

    HSS benchmarks intuitive visual reasoning with human-authored image and video tasks organized by a structured taxonomy and evaluated against human-annotated atomic rubrics.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    Across HSS, frontier models remain well below humans: GPT-6-astra achieves a 53.6% pass rate versus 93.1% for human participants, even at maximum reasoning effort.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HSS establishes intuitive visual reasoning as a measurable axis of multimodal intelligence where current scaling paradigms have yielded limited progress.

    Source: §6 · p. 10

    HSS excludes video items whose resolution depends on speech or off-screen sound because it supplies neither audio nor transcripts.

    Source: §5 · p. 9

Abstract

from arXiv · show

Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit between two parked cars; a few seconds of video reveals who holds authority in a room; and a fleeting clip highlights subtle abstract patterns like unwritten rules or hidden labels. This capacity reflects a form of humanity's sixth sense: an intuitive reasoning mechanism that recovers implicit information beyond raw sensory perception. Crucially, this rapid, zero-shot visual intuition underpins everyday navigation and social interaction, making it a vital capability for Multimodal Large Language Models (MLLMs) deployed alongside people. Existing visual benchmarks, however, target either deliberate expert-level analysis in academic and mathematical domains or low-level perception, leaving the intuitive reasoning that people perform largely untested. To bridge this gap, we introduce Humanity's Sixth Sense (HSS), a benchmark for intuitive visual reasoning. HSS spans diverse image and video inputs, organizes items under a structured taxonomy, and pairs each with human-written prompts probing the implicit temporal, spatial, social, and abstract structure that people infer at a glance. Frontier MLLMs fall short of human performance: participants reach 93.1% accuracy, while the strongest model, GPT-6-astra, reaches only 53.6% even at maximum reasoning effort. Despite excelling in many complex tasks that require advanced perception and knowledge, current models still struggle significantly on these visual tasks that are intuitive for humans. We further explore agentic setup that apply dynamic visual manipulation to HSS, which narrows but does not close the gap. HSS establishes intuitive visual reasoning as a measurable axis and directs attention to a capability that scaling on current benchmarks has so far left behind.

1 INTRODUCTION

Existing visual benchmarks emphasize specialized problem-solving or narrow perception, leaving intuitive inference of implicit scene information under-tested. HSS targets this gap, but frontier models remain well below human performance.

1 INTRODUCTION

  • Existing visual benchmarks emphasize specialized academic problem-solving, low-level perception, or fragmented probes, rather than inferring implicit scene information.HSS distinguishes this target from benchmarks that test what is explicitly present or rely on narrow, synthetic, or multiple-choice tasks.Source: §1 · p. 2
  • HSS uses open-ended image and video tasks across four domains and eleven subdomains, with human-written prompts and human-annotated atomic rubrics.The domains are temporal and causal dynamics, physical and spatial logic, social understanding, and abstract contextual inference.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • 53.6% pass rate for GPT-6-astra versus 93.1% for humans reveals a substantial HSS gap, with standard scaling and agentic strategies yielding only modest improvements.GPT-6-astra reaches 53.6% even at maximum reasoning effort.Source: §1 · pp. 2–3
  • The results frame intuitive visual reasoning as a distinct, largely unexplored dimension that does not emerge simply from scaling test-time compute.The paper also examines reasoning effort and agentic tool-use workflows, finding the capability gap resistant to both.Source: §1 · p. 3
  • Human-authored instances pass through multi-stage independent review to address the subjectivity of intuitive judgments and establish human consensus.The paper presents this curation as a way to ensure high data precision.Source: §1 · p. 3

2 THE HSS BENCHMARK

HSS turns intuitive visual inference into a free-response benchmark with atomic scoring criteria, a four-domain taxonomy, explicit curation principles, and multi-stage human review.

2.1 TASK FORMAT

  • Each task pairs media and a question with a human-authored reference answer and atomic rubrics; an answer is solved only when every criterion is met.The judge marks each rubric criterion as met or unmet.Source: §2.1 · pp. 3–4

2.2 HSS TAXONOMY DESIGN

  • The taxonomy organizes intuitive inference into four domains and eleven subdomains, serving both as a curation blueprint and a fine-grained diagnostic framework.Its domains cover temporal and causal dynamics, physical and spatial logic, social understanding, and abstract contextual inference.Source: §2.2 · p. 4

2.3 TASK DESIGN PRINCIPLES

  • Tasks must require inference beyond explicit perception, avoid specialized expertise, and command unambiguous human consensus.Tasks are rejected when a high-resolution crop makes the answer obvious, or when independent reviewers do not unanimously agree on the ground truth.Source: §2.3 · p. 4

2.4 BENCHMARK CONSTRUCTION PIPELINE

  • 522 of 3,466 authored tasks survive three unanimous review rounds, yielding a 15.1% acceptance rate and a benchmark of 288 image and 234 video instances.The released benchmark spans four domains and eleven subdomains and contains 723 rubric criteria; three reviewers beyond the author verify each admitted rubric.Source: §2.4 · pp. 4–5

3 EXPERIMENTS

The experiments evaluate multimodal models and people on HSS, finding a large performance gap and systematic failure patterns. Agentic inspection improves some results, while reasoning-effort and judge analyses probe the scope and robustness of those findings.

3.1 BENCHMARK RESULTS

  • 7.3 percentage points: video tasks average lower than static images for 23 of 25 models; social understanding is lowest for 21 of 25, averaging 24.4% versus 34.1% across the other domains.Without visual media, the top model’s score falls from 53.6% to 6.6%.Source: §3.1 · p. 5
  • 53.6%: GPT-6-astra at maximum reasoning effort trails human participants’ 93.1% pass rate on HSS.Even annotators who reported uncertainty scored 85.7% on those tasks.Source: §3.1 · pp. 5–6, Table 1 · p. 6

3.2 MODEL FAILURE ANALYSIS

  • 53% of failures stem from perception and 41% from latent inference, while abstract reasoning accounts for 5%; missed cues and object or role misidentification are the largest individual causes.Failure causes vary by subdomain, and on tasks failed by at least five models, a median 80% share the same cause.Source: §3.2 · p. 6, Figure 3 · p. 7

3.3 AGENTIC HARNESSES

  • Across 388 tasks, all four harness/base pairs improve accuracy, but Codex with GPT-6-astra reaches only 59.3%.Reinspection reduces perception and latent errors but not depth errors; gains peak in patterns & pareidolia (+30.6 points) and retrodiction (+18.9). Failed runs take a median 81 versus 67 steps for successful runs, and loops sometimes overanalyze correct initial readings.Source: §3.3 · p. 7, Figure 4 · p. 8

3.4 ABLATION STUDIES

  • Higher reasoning effort usually improves overall performance but can lower subdomain accuracy; GPT-6-astra’s retrodiction accuracy drops 14 points from high to xhigh.Alternative judges closely track the primary judge (r ≥0.98), with over 95% agreement at both item and criterion levels; GPT-5.6-terra is uniformly stricter by 2.4 points on average.Source: §3.4 · pp. 8–9, Figure 5 · p. 8, Figure 6 · p. 9

4 RELATED WORK

HSS is positioned as a benchmark for implicit visual inference, distinct from both visual problem-solving and perception benchmarks. It brings several reasoning areas together over natural media under a shared admission standard.

4 RELATED WORK

  • Unlike visual problem-solving and perception benchmarks, HSS asks for answers not transcribable from the image or discoverable by looking harder.Its tasks target what was never depicted in the frame.Source: §4 · p. 9
  • Related benchmarks cover parts of visual abduction or spatial, physical, social, and abstract reasoning, while HSS spans these inferences over natural media under one admission standard.HSS admits items only when reviewers agree that the answer is forced rather than merely plausible.Source: §4 · p. 9

5 LIMITATIONS

The authors identify limits involving the visual-only input channel, LLM-mediated scoring, and possible prior exposure to public-web media.

5 LIMITATIONS

  • Video items omit audio and transcripts, so tasks requiring speech or off-screen sound are outside HSS’s scope.The benchmark evaluates what models infer from visual input alone.Source: §5 · p. 9
  • LLM-judge scores may reflect judge biases, including self-preference, although cross-judge agreement and an oracle condition bound the reported effect.The authors report cross-judge agreement and score an oracle condition using human reference answers.Source: §5 · p. 9
  • Because media comes from the public web, prior exposure to an individual image or clip cannot be fully excluded, despite newly authored questions.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents HSS as a measurable benchmark of intuitive visual reasoning and reports a persistent gap between frontier models and people. The related-work appendix situates HSS among benchmark families and adjacent evaluation practices.

6 CONCLUSION

  • HSS measures inference beyond explicitly depicted pixels, and frontier models remain far behind people despite test-time compute scaling and agentic harnesses.The conclusion frames intuitive visual reasoning as a measurable axis where current scaling paradigms have yielded limited progress.Source: §6 · p. 10
  • Benchmark tasks and media are human-authored and reviewed, while released media is restricted to specified redistributable licenses with attribution and local copies.The authors state that AI assistants were used only to polish drafts and assist with evaluation and analysis code.Source: AI USE STATEMENT · p. 10, ETHICS STATEMENT · p. 10
  • Theory-of-mind results measure agreement with annotator judgments, not ground truth about depicted people, because interpretations are culturally situated.Consensus among a demographically narrow annotator pool does not establish a universal inference.Source: ETHICS STATEMENT · p. 10
  • The authors plan to release evaluation prompts and controls, while confirmatory metrics use a frozen versioned benchmark of 522 tasks.Media are content-addressed and cached so models receive byte-identical inputs.Source: REPRODUCIBILITY STATEMENT · p. 10

A EXTENDED RELATED WORK

  • Academic visual problem-solving benchmarks preserve tasks under verbal image descriptions, whereas perception benchmarks ask models to locate answers already in the frame.HSS is presented as complementary to these benchmark families because its answers are not in the frame.Source: A EXTENDED RELATED WORK · p. 14
  • Visual abduction benchmarks probe intentions, prior events, or continuations, while HSS combines four inference areas and requires reviewer agreement that answers are forced.This admission standard supports open-ended rubric grading.Source: A EXTENDED RELATED WORK · p. 14
  • Prior spatial, physical, social, and abstract reasoning suites often use synthetic or simulated scenes; HSS extends these areas to natural media and unobserved inferences.Source: A EXTENDED RELATED WORK · p. 14
  • The agentic condition tests whether active inspection repairs HSS failures, while automated open-ended grading is audited across judges for agreement and self-preference.The related-work discussion presents this agentic condition as diagnostic rather than a method contribution.Source: A EXTENDED RELATED WORK · p. 14

B BENCHMARK DETAILS

The benchmark details describe the corpus composition, its authoring and review attrition, and the taxonomy used to organize intuitive visual reasoning.

B.1 COMPOSITION AND STATISTICS

  • Media types and scene coverage vary by domain: theory of mind is almost entirely video, while spatial subdomains lean on images.The corpus spans diverse scene types and video durations; Figure 7 summarizes category breakdowns, domain coverage, and clip lengths.Source: §B.1 · p. 15, Figure 7 · p. 15

B.3 TAXONOMY DESIGN

  • The taxonomy partitions intuitive inference into temporal and causal dynamics, physical and spatial logic, social understanding, and abstract and contextual inference.Its categories draw on distinctions among core knowledge of objects and space, intuitive physics, and mentalizing.Source: §B.3 · p. 15

B.1 COMPOSITION AND STATISTICS

  • The released HSS benchmark contains 522 tasks across four domains and eleven subdomains, with 723 rubric criteria and 288 images plus 234 video clips.The video totals 17.6 hours, with a median duration of 76 seconds and a longest clip of 28 minutes.Source: §B.1 · p. 14, Table 3 · p. 16

B.2 CONSTRUCTION PIPELINE

  • Of 3,466 authored tasks, 522 pass all three review rounds, a 15.1% yield; the first gate rejects 2,302, often for depicted answers or ambiguous ground truth.Returned tasks re-enter authoring and are counted at the round where they finally pass.Source: §B.2 · p. 15, Figure 8 · p. 16

C EXPERIMENTAL SETUP

The setup standardizes model calls and media handling, uses rubric-based judging and pass@1 scoring, and tests whether the benchmark can be solved without visual input. Prompts distinguish native video from sampled frames and give judges fixed grading instructions.

C EXPERIMENTAL SETUP

C.1 MODEL CONFIGURATIONS

  • Model configurations specify endpoints and request settings; temperature and top-p use provider defaults, and reasoning effort is set to the highest level each deployment honors when possible.The main table uses the listed effort, and requested output-token ceilings include thinking tokens.Source: §C.1 · p. 17, Table 5 · p. 18

C.2 MEDIA HANDLING

  • Native-video models receive clips directly, while other models receive uniformly sampled frames capped at 500 or a stricter provider limit.When a clip exceeds the cap, the sampling rate is reduced rather than truncating the clip; frames are timestamped, and media is cached byte-identically.Source: §C.1 · p. 17, §C.2 · pp. 17–18, Table 5 · p. 18

C.3 JUDGING

  • A single LLM judge evaluates each criterion against the question, reference answer, rubric, and candidate response, without seeing the media by default.The protocol gives credit only when the candidate explicitly asserts required content, treats the reference as ground truth on disagreement, and records bounded unparseable-output retries as grading failures.Source: §C.3 · pp. 18–19, §C.6 · p. 20

C.4 METRICS

  • Pass@1 counts an attempt as correct only when every rubric criterion is met, then averages per-sample success across tasks.Each main-table sample has three attempts except where provider errors leave fewer usable attempts; effort and frame-budget sweeps use one attempt.Source: §C.4 · p. 19

C.5 THE BLIND CONTROL

  • With media removed, GPT-6-astra at maximum effort scores 6.6% versus 53.6% sighted, and only 4 of 522 tasks pass blind on all three attempts.The 67 tasks passing at least once are described as unstable, chance-like successes.Source: §C.5 · p. 19

C.6 PROMPTS

  • The prompt set includes image and video instructions, with sampled-frame prompts specifying frame count, rate, duration, timestamps, and chronological order.The blind-control prompt asks the model to answer faithfully and say when information is insufficient rather than guess.Source: §C.6 · pp. 19–20

D FAILURE ANALYSIS

The failure analysis labels failed attempts by where the answer goes wrong and finds that causes are systematic across tasks and media types. Agentic re-inspection repairs many failures, but not the depth error of treating a 2D overlap as 3D alignment.

D FAILURE ANALYSIS

  • Perception and latent causes together account for 94% of the pooled failures across 24 models.The analysis labels 8,573 failed first attempts, grouping eleven causes into perception, latent inference, and reasoning families, with a twelfth label for no committed answer.Source: D FAILURE ANALYSIS · pp. 20–21, Table 6 · p. 21
  • On tasks where five or more models fail, a median 80% share the same failure cause, and cause is strongly associated with subdomain and media type.The per-task view also finds that a median 90% share the same cause family.Source: D FAILURE ANALYSIS · p. 21
  • Image failures are 66% perception errors, while video failures are 57% latent errors, including a 28% share from missing the decisive cue.Image errors commonly involve orientation, object identification, size and fit, or reading 2D overlap as 3D alignment.Source: D FAILURE ANALYSIS · p. 21
  • Agentic re-inspection reduces most perception and latent failure causes by 19–37%, but 2D-overlap-to-3D-alignment errors barely change, from 102 to 101.Only 24% of single-call failures in that category are repaired, the lowest fix rate among common perception causes.Source: D FAILURE ANALYSIS · p. 21

E FAILURE EXAMPLES

  • Figures 9–24 present failure examples alongside the media, prompt, model answer, and human-written reference answer.The examples cover hidden or invisible information, spatial reasoning, temporal inference, social reasoning, and abstract patterns.Source: E FAILURE EXAMPLES · p. 21

F AGENTIC HARNESSES ANALYSIS

Agentic harnesses improve performance over single calls, with gains varying by harness and by the evidence available in the media. Example trajectories show that extra inspection can recover brief visual cues, but can also reinforce an incorrect interpretation.

F AGENTIC HARNESSES ANALYSIS

  • Across Claude Code trajectories, tasks answered incorrectly take a median 81 versus 67 steps around Opus-5 and 65 versus 54 around Fable-5.1; Codex shows no such gap.The comparison is between failures and successes; Codex searches least.Source: F AGENTIC HARNESSES ANALYSIS · p. 38

F.1 FOUR TRAJECTORIES

  • In the repaired rugby example, repeated sampling and close inspection recovered a brief post-whistle celebration cue that the single call misread.The agentic answer identified the white team as the winner; the single call incorrectly favored the dark red-and-black team.Source: §F.1 · pp. 38–39

F AGENTIC HARNESSES ANALYSIS

  • Across four harness/base pairs on 388 paired tasks, all setups improve performance: Claude Code gains 16–21 points at 12–14× the base model’s thinking tokens, while Codex gains 5–8 points at 1–2×.Table 9 reports the paired harness/base comparisons.Source: F AGENTIC HARNESSES ANALYSIS · p. 21, Table 9 · p. 39

F.1 FOUR TRAJECTORIES

  • In a broken spatial example, precise image measurements led the agent to treat a 2D overlap as 3D alignment and overrule its own depth caveat.The single call passed, while the agentic answer failed; the example mirrors the depth-related cause that the agent loop did not reduce.Source: §F.1 · p. 39
  • In a broken inspection example, the agentic answer omitted the workers’ hand-back action despite sampling the relevant video window and revising its answer.The single-call answer passed; the agentic answer described inspection but not the return required by the reference.Source: §F.1 · p. 40

G REASONING EFFORT IN FULL

Higher reasoning effort often raises domain-level accuracy, but gains differ by model and domain, and maximum effort is not consistently best. Several model–subdomain cells end at or below their starting accuracy.

G REASONING EFFORT IN FULL

  • From lowest to highest effort, GPT-6-astra gains +8.2 temporal, +12.5 physical, +15.5 social, and +7.8 abstract; Claude-Opus-5.5 gains +6.7, +9.1, +0.9, and +4.9, respectively.Gemini-3.8-flash changes by +0.7 temporal, +5.7 physical, +3.6 social, and −1.0 abstract.Source: G REASONING EFFORT IN FULL · p. 40
  • Seven of 33 model × subdomain cells finish no better than they started, and three of Gemini-3.8-flash’s eleven finish lower.Source: G REASONING EFFORT IN FULL · p. 40
  • Only five of the twelve model × domain curves achieve their best score at the highest available reasoning setting.Source: G REASONING EFFORT IN FULL · p. 40

H VIDEO FRAME BUDGET IN FULL

The video-frame sweep tests whether denser temporal sampling improves performance across 234 HSS video tasks. Gains diminish beyond 100 frames, and the available intervals do not resolve whether small further gains are real.

H VIDEO FRAME BUDGET IN FULL

  • Only GPT-6-astra’s end-to-end gain from 10 to 500 frames has non-overlapping endpoint intervals; beyond roughly 100 frames, the sweep cannot resolve small gains.The authors read the results as ruling out a large monotone gain from temporal resolution beyond roughly 100 frames, but not a small one.Source: H VIDEO FRAME BUDGET IN FULL · p. 40, Table 11 · p. 41
  • From 10 to 100 frames, the four models gain +11.5, +6.0, +2.1, and +9.8 points; from 100 to 500, they gain +6.4, +2.1, +3.8, and −1.3 points.The latter step uses five times as many tokens, and per-cell intervals are about ±6 points.Source: H VIDEO FRAME BUDGET IN FULL · p. 40
  • The sweep compares Pass@1 on 234 video tasks across frame caps, with 2-fps sampling thinned uniformly so every budget spans the whole clip.The 500-frame configuration is used elsewhere in the paper; this sweep reports first attempts alone.Source: H VIDEO FRAME BUDGET IN FULL · p. 40, Table 11 · p. 41

I JUDGE ALIGNMENT IN FULL

Three vendor-separated judges largely agree on model ordering, although their score levels differ. The study also finds no detectable evidence of same-vendor judging effects at the resolution HSS supports.

I JUDGE ALIGNMENT IN FULL

  • Five leaderboard models were re-graded end to end by three judges from different vendors using identical rubrics, reference answers, and grading prompts.Per-model scores under each judge are reported in Table 12.Source: I JUDGE ALIGNMENT IN FULL · p. 41, Table 12 · p. 42
  • All three judges preserve the same model ordering (ρ = 1.0 for every pair), while GPT-5.6-terra scores models 2.4 points below the other judges’ mean.Claude-Opus-5 and Gemini-3.6-flash differ by 0.3 points on average.Source: I JUDGE ALIGNMENT IN FULL · p. 42, Table 12 · p. 42
  • Pairwise agreement reaches 95.8–97.2% at criterion level and 96.2–97.5% at item level; all three judges give the same verdict on 95.2% of items.Cohen’s κ is 0.914–0.946 on both units.Source: I JUDGE ALIGNMENT IN FULL · p. 42
  • The four same-vendor residuals are small, with two negative; the authors find no detectable vendor-bias evidence at HSS’s current resolution, not proof that bias is absent.Residuals are calculated after subtracting each judge’s baseline leniency from same-vendor interaction effects.Source: I JUDGE ALIGNMENT IN FULL · p. 42
Loading 2610.08966v2…