PaperlayerView exact revision on arXiv

Source-linked AI summary

Perception or Prejudice: Can MLLMs Go Beyond First Impressions of Personality?

Caixin Kang, Tianyu Yan, Sitong Gong, Mingfang Zhang, Liangyang Ouyang, Ruicong Liu, Bo Zheng, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang

arXiv:2605.22109v1cs.AIcs.CVcs.CY

TL;DR

Existing MLLM personality benchmarks score Big Five predictions without testing whether models use supporting behavioral evidence. The paper introduces Grounded Personality Reasoning and MM-OCEAN, then benchmarks the full rating–reasoning–grounding chain. Across 27 MLLMs, 51% of correct ratings lack grounded evidence and mean Holistic-Grounding Rate is only 10.4%.

  • Problem

    Existing apparent-personality benchmarks use numerical Big Five prediction without evaluating whether ratings are grounded in observable behavioral evidence.

  • Method

    The paper introduces Grounded Personality Reasoning and MM-OCEAN, combining evidence-linked trait judgments with three-tier rating, reasoning, and grounding evaluation.

  • Results

    51% of correct ratings lack grounded evidence, while mean Holistic-Grounding Rate is only 10.4% across 27 evaluated MLLMs.

  • Takeaways & Limitations

    Traditional rating-only evaluations can overestimate personality-assessment competence by crediting ungrounded predictions.

  • Takeaways & Limitations

    MM-OCEAN focuses on apparent personality in short, single-speaker English video clips and evaluates Task 2 reasoning with an AI-as-Judge protocol.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) are increasingly deployed in human-facing roles where personality perception is critical, yet existing benchmarks evaluate this capability solely on numerical Big Five score prediction, leaving open whether models truly perceive personality through behavioral understanding or merely prejudge through superficial pattern matching. We address this gap with three contributions. (i) A new task: we formalize Grounded Personality Reasoning (GPR), which requires MLLMs to anchor each Big Five rating in observable evidence through a chain of rating, reasoning, and grounding. (ii) A new dataset: we release MM-OCEAN (1,104 videos, 5,320 MCQs), produced by a multi-agent pipeline with human verification, with timestamped behavioral observations, evidence-grounded trait analyses, and seven categories of cue-grounding MCQs. (iii) Benchmark and analysis: we design a three-tier evaluation (rating, reasoning, grounding) plus four sample-level failure-mode metrics: Prejudice Rate (PR), Confabulation Rate (CR), Integration-failure Rate (IR), and Holistic-grounding Rate (HR), and benchmark 27 MLLMs (13 closed, 14 open). The analysis uncovers a striking Prejudice Gap: across the field, 51% of correct ratings are not grounded in retrieved cues, and the Holistic-Grounding Rate spans only 0-33.5%. These findings expose a disconnect between getting the right score and reasoning for the right reason, charting a roadmap for grounded social cognition in MLLMs.

1 Introduction

Personality perception is increasingly important for human-facing MLLM applications, but numerical Big Five benchmarks cannot distinguish genuine behavioral understanding from superficial pattern matching. The paper introduces grounded evaluation, a verified benchmark, and failure-mode analysis to expose this gap.

  • Motivation: MLLMs support applications including interview screening, mental-health triage, adaptive social robots, and affect-aware game characters.These systems infer stable psychological characteristics from observable behavior, with Big Five traits as the target.
  • Problem: Existing apparent-personality benchmarks use numerical Big Five regression, so models can achieve accurate predictions through superficial correlations rather than supporting behavioral evidence.For example, smiling faces may be associated with high agreeableness without genuine cue understanding.
  • Contributions: Grounded Personality Reasoning (GPR) requires fine-grained cue perception, evidence-based trait reasoning, and structured probes of sub-skills such as localization and temporal-causal reasoning.A personality judgment is treated as trustworthy only when grounded in observable behavioral evidence.
  • Benchmark and analysis: The evaluation combines rating, reasoning, and grounding tasks with Prejudice, Confabulation, Integration-failure, and Holistic-Grounding rates.These metrics analyze where the personality-reasoning chain breaks at the sample level.
  • Findings: 51% of correct ratings lack grounded cue retrieval, while Holistic-Grounding Rate spans 0–33.5% across 27 MLLMs.The benchmark therefore distinguishes correct scores from reasoning supported by the right evidence.
  • Contributions: MM-OCEAN contains 1,104 videos and 5,320 MCQs with timestamped observations, evidence-grounded analyses, and seven cue-grounding categories.The benchmark is produced through a multi-agent pipeline with human verification.

2 Related Work

MM-OCEAN builds on Big Five apparent-personality recognition and multimodal social-cognition benchmarks, while targeting a previously unsupported combination of personality inference, grounded reasoning, chain evaluation, and fine-grained cue grounding.

  • Big Five and APR: The Big Five model is the adopted target of inference in MM-OCEAN and is widely used as an empirically supported personality taxonomy.Prior apparent-personality recognition work also uses Big Five scores.
  • Big Five and APR: Existing apparent-personality benchmarks remain numerical regression tasks and provide no mechanism to evaluate why a score was assigned.GPR reframes the task around behaviorally grounded reasoning rather than numerical outputs alone.
  • Video understanding: Video-understanding benchmarks cover temporal reasoning, long-form comprehension, and multimodal tasks, with some attention to emotion recognition or action detection.They do not jointly target personality, grounded reasoning, reasoning-chain evaluation, and fine-grained cue probes.
  • Social cognition: Theory-of-mind benchmarks study momentary mental states, whereas this work addresses stable personality traits through multimodal integration over longer time spans.The paper positions personality perception as a higher-order social-cognitive task.

3 MM-OCEAN: Benchmark Construction

MM-OCEAN is a multimodal benchmark whose GPR instances connect video inputs, Big Five trait ratings, evidence-linked reasoning chains, and seven cue-grounding MCQ categories. Its five-stage construction pipeline combines LLM agents, human verification, automated checks, leakage filtering, and expert review.

  • Task and data representation: A GPR instance contains RGB frames, audio, transcription, five Big Five traits, and a five-level ordinal personality scale.The input is represented as V=(Vvis, Vaud, Vtxt), with trait levels ranging from Very Low to Very High.
  • Task and data representation: Each reasoning chain contains a trait level, cited observation indices, and a rationale, while each video has seven cue-grounding MCQs.The evidence constraint distinguishes GPR from APR, which evaluates only predicted trait scores.
  • Construction pipeline: MM-OCEAN uses four LLM agents and human roles comprising 24 trained annotator-verifiers and expert reviewers in a five-stage pipeline.The pipeline interleaves observation, trait reasoning, MCQ generation, alignment, and quality review.
  • Construction pipeline: Human annotators verify atomic observations, refine timestamps, and add bounding boxes for retained Expression and Action cues.Observer drafts were accepted at 78.2%, corrected at 14.6%, and deleted at 5.9%.
  • Construction pipeline: The Psychologist produces structured Big Five analyses that map trait scores to five ordinal levels and cite verified cues as evidence.Each analysis includes a confidence-weighted rationale.
  • Construction pipeline: The Examiner generates seven MCQ categories spanning semantic or causal reasoning and pixel- or time-level visual grounding.The reasoning cluster includes personality attribution, counterfactual reasoning, temporal-causal chains, and mixed-emotion discrimination.
  • Quality control: The Aligner checks timestamps, bounding boxes, semantic consistency, and factual alignment, after which text-leakage filtering and expert review remove transcript-derivable items.The benchmark draws from short single-speaker clips in ChaLearn First Impressions V2.
  • Dataset statistics: The released benchmark contains 1,104 videos, approximately 13.5K verified observations, 5,520 trait analyses, and 5,320 retained MCQs.Observations span Expression, Action, Audio, and Background channels.

4 Evaluation Framework

The evaluation measures personality reasoning at increasing depth through rating, explanation, and cue-grounding tasks, then combines these outcomes into population-level alignment signals and sample-level failure rates. Threshold robustness supports the stability of Holistic-Grounding rankings.

  • Three-tier evaluation: MM-OCEAN evaluates ordinal personality rating, open-ended rating reasoning, and structured cue grounding as three tasks of increasing cognitive depth.Cross-task diagnostics localize where the personality-reasoning chain breaks.
  • Task 1: Ordinal Personality Rating: Task 1 reports exact-match accuracy and mean absolute error for each predicted ordinal Big Five level.Spearman’s ρ is reported in the appendix.
  • Task 2: Rating Reasoning: Task 2 evaluates explanations using AI-as-Judge scores for Evidence Coverage, Logical Coherence, Grounding Accuracy, and Directional Accuracy.Each dimension receives a score from 1 to 10.
  • Task 3: Structured Cue Grounding: Task 3 measures grounding through structured MCQs covering seven cue-grounding categories, with overall and per-category accuracy.The task isolates whether judgments are linked to specific observable cues.
  • Cross-task diagnosis: Rating–Grounding Misalignment compares a model’s average T2/T3 rank with its T1 rank, where positive values indicate weaker downstream grounding relative to rating.The closed-Tk signal provides an ecosystem-level snapshot of task performance.
  • Failure modes: The four sample-level rates identify right-rating/wrong-cue, right-rating/incoherent-reasoning, right-cue/wrong-rating, and fully correct outcomes.Default thresholds are θ1=θ3=0.5 and θ2=0.7.
  • Robustness: HR rankings remain stable under a 3×3×3 threshold sweep, with ρ≥0.92 across all 27 threshold combinations.PR, CR, and IR are minimized, whereas HR captures complete three-tier success.

5 Benchmarking Results

Across 27 MLLMs, personality rating and verbal reasoning outperform fine-grained cue retrieval, producing substantial grounding gaps and distinct failure archetypes.

  • 5.2 Leaderboard and the Prejudice Gap: 51.3% mean Prejudice Rate and 10.4% mean Holistic-Grounding Rate show that many correct ratings lack grounded cue retrieval.HR ranges from 0.0% to 33.5%, with Gemini 3 Flash reaching the field’s best value.
  • 5.2 Leaderboard and the Prejudice Gap: Top-3 frontier models differ little on rating and explanation but substantially on cue retrieval, with gaps of −5.6%, −3.6%, and −26.6% respectively.Top-3 mean PR is approximately 14.5% for proprietary models and 47.0% for open-source models.
  • 5.3 Where Prejudice Concentrates: Cognitive and Per-Sample Diagnostics: 64.8% accuracy makes Temporal-Causal Reasoning easiest, while Spatial Localization at 30.7% and Micro-expression Localization at 34.6% are hardest.Closed-source advantages concentrate on visual grounding rather than reasoning categories.
  • 5.3 Where Prejudice Concentrates: Cognitive and Per-Sample Diagnostics: +19.5 pp and +21.8 pp are the Top-3 closed advantages on Spatial Localization and Temporal-Spatial Joint, compared with 6–11 pp on reasoning-cluster categories.Even Gemini 3.1 Pro reaches only 57% on Spatial Localization and 71% on Temporal-Spatial Joint.
  • 5.3 Where Prejudice Concentrates: Cognitive and Per-Sample Diagnostics: HR has greater model-discriminating variation than any single-task metric while remaining strongly correlated with the equally weighted task mean.HR’s coefficient of variation is approximately 0.93, versus approximately 0.13, 0.16, and 0.36 for T1, T2, and T3.
  • 5.3 Where Prejudice Concentrates: Cognitive and Per-Sample Diagnostics: Confident Raters perform well on T1 but poorly downstream, whereas Cautious Reasoners show the opposite pattern; 17 remaining models occupy the balanced middle band.The archetypes are defined by RGM thresholds of ≥+5 and ≤−5, respectively.

6 Discussion and Conclusion

The paper concludes that grounded personality reasoning is needed because rating-only evaluation overestimates MLLM competence. It identifies cue retrieval as a central frontier while acknowledging scope and evaluation limitations.

  • 6 Discussion and Conclusion: 51% of correct ratings lack grounded evidence, while mean HR is only 10.4%, revealing a disconnect between correct scores and grounded reasoning.The conclusion frames HR as a highly discriminative metric and reports a −26.6% proprietary/open-source cue-retrieval gap.
  • 6 Discussion and Conclusion: GPR and MM-OCEAN require MLLMs to ground personality judgments in observable evidence through a multi-granularity benchmark.The benchmark evaluates rating, reasoning, and grounding rather than numerical outputs alone.
  • 6 Discussion and Conclusion: Fine-grained spatiotemporal grounding is identified as essential for developing trustworthy, personality-aware MLLMs.This conclusion follows the reported weakness in cue retrieval relative to rating and explanation.
  • Limitations and future work: MM-OCEAN focuses on apparent personality from short, single-speaker English video clips tied to the First Impressions V2 construct.The authors propose cross-cultural and multilingual videos as natural extensions.
  • Limitations and future work: Task 2 reasoning quality is evaluated through an AI-as-Judge protocol, with multi-judge ensembles proposed to improve reliability.The paper also calls for richer grounding operationalizations beyond MCQ-based cue retrieval.

B Human Annotation Protocol

MM-OCEAN’s human annotation protocol combines trained verification, frame-precise cue editing, bounding-box refinement, and quality monitoring to produce reliable behavioral evidence.

  • Annotation workflow: 24 trained annotators verified Observer-drafted behavioral cues using a three-view tool for video scrubbing, cue editing, and bounding-box annotation.The tool supported frame-level timestamp refinement, dimension and body-part labels, and visual localization of cues.
  • Quality control: Each retained observation received a verifier decision and, for Expression or Action cues, tightened boxes and perceptually meaningful start and end frames.The three-way decision was correct, incorrect and reworded, or nonexistent and deleted.
  • Annotation workflow: 78.2% of 45,609 drafted clues were accepted, 14.6% corrected, 5.9% deleted, and 605 missed cues added by annotators.Corrections covered descriptions, timestamps, or dimensions; deleted cues were nonexistent or hallucinated.
  • Quality control: Quality monitoring filtered approximately 8% of submitted videos, leaving core annotators with a 75.7% mean acceptance rate and 23% correction-plus-deletion rate.Abnormally high acceptance rates triggered review and removal.
  • Quality control: 77.0% pairwise verdict agreement across 147 independently annotated video pairs indicates reproducibility of the three-way cue assessment.Agreement ranged from 50% to 100% across paired videos.
  • Evaluation implications: Task 1 uses five ordinal Big Five levels, while exact-match accuracy is emphasized because approximately 80% of errors are off by one level.This error concentration makes mean absolute error weakly discriminative.

E Task 2 Logical-Coherence Distribution

Task 2 Logical-Coherence scores concentrate near the upper-middle range for top models and lower-middle range for weaker models, with maximal scores nearly absent; the benchmark also contains a small human-only question subset.

  • Logical-Coherence distribution: Top models concentrate around the 8–9 Logical-Coherence bucket, whereas weaker models concentrate in the 4–5 bucket.The distribution separates model groups without suggesting score inflation.
  • Logical-Coherence distribution: Only LLaVA-NeXT reaches the 10/10 Logical-Coherence bucket, at 0.4% of samples.Virtually no model reaches the maximum score.
  • Question difficulty: 153 questions, or 1.8% of MM-OCEAN, are answered correctly by no evaluated model.The full per-question correctness distribution is roughly bell-shaped with a long left tail.

G Additional Model-Level Analyses

Additional analyses examine relative cue-grounding expertise, parameter efficiency, substitutability, question difficulty, and concrete multimodal reasoning examples across models.

  • Relative expertise: Every evaluated MLLM shows positive relative deviation on the reasoning cluster and negative deviation on the visual-grounding cluster.The centered expertise vector subtracts each model’s own seven-category mean to isolate relative strengths.
  • Reasoning quality: Top models concentrate about 50% of Task 2 samples in the 8–9 Logical-Coherence bucket, while weaker models concentrate in 4–5.Virtually no model reaches the 10/10 bucket.
  • Parameter efficiency: MiMo-VL-7B-RL is the most parameter-efficient open-source model at 0.315 B per %-T3-above-chance.The analysis reports larger parameter costs for heavy models and states that data quality and post-training appear more important than parameter count.
  • Open-source substitutability: T1 is fully substitutable with a gap below 2 percentage points, T2 is partially substitutable with a gap of approximately 0.2 points, and T3 remains the bottleneck.Gemma-4-31B-it reaches 57.0% on T3 versus 56.5% for the proprietary Flash class, while no open model reaches Flash-Pro’s 65+% tier.
  • Grounded reasoning examples: The worked examples test trait attribution, counterfactual rating changes, temporal-causal ordering, emotional-state characterization, and spatial-temporal cue localization.Examples include a rapport-building gesture supporting Agreeableness and an ordered look-away–pause–assertive re-engagement chain.
  • Question difficulty: 153 questions, or 1.8%, are answered correctly by no model, forming a small human-only subset of MM-OCEAN.Question-level correctness across models follows a bell-shaped distribution with a long left tail.
  • Robustness: A 3×3×3 threshold sweep preserves stable HR rankings with Spearman ρ∈[0.925, 1.000] across threshold combinations.The Top-3 closed and open identities are preserved for 21 of 27 combinations; rankings change when the Judge threshold is set to 8/10.

J AI-as-Judge Protocol

The AI-as-Judge protocol scores Task 2 reasoning against verified evidence on four dimensions, with robustness checks indicating that scores track correctness and yield stable rankings across judges.

  • Protocol: GPT-4o-mini evaluates Task 2 outputs at temperature 0 using a single standardized judge across all 27 models.The judge was selected outside the leaderboard’s high end to reduce self-preference concerns.
  • Protocol: The judge sees each model output, the ground-truth trait level, and human-verified observations, but not the video itself.This defines the reference information available for evaluation.
  • Robustness checks: When Task 1 was wrong, Judge scores fell by approximately 2.1–3.4 points across all 27 models.The conditional comparison tests whether the judge follows correctness rather than merely response style.
  • Robustness checks: Alternative judges produce ST2 rank correlations of ρ=0.94 and ρ=0.92 with GPT-4o-mini.The alternative judges were Claude Haiku 4.5 and Gemini, evaluated on a stable 200-video subset.
  • Caveat: GPT-4o-mini scores its own GPT family approximately +1.0 point above the cross-family average, but relative rankings remain correlated at ρ≥0.91.The observed self-preference is described as modest absolute inflation rather than ranking distortion.

K Dataset Documentation

MM-OCEAN documents a research benchmark built from verified behavioral annotations, trait analyses, and cue-grounding questions, with explicit scope, use, and validity caveats. Its analyses show that trait difficulty, especially for Neuroticism, and cue-grounding performance remain uneven across models and parameter scales.

  • Composition: MM-OCEAN contains 1,104 videos, approximately 13.5K verified observations, 5,520 trait analyses, and 5,320 cue-grounding MCQs.Each 15-second clip includes observations across Expression, Action, Audio, and Background, plus five trait analyses and seven MCQ categories.
  • Evaluation documentation: Task 3 performance is evaluated by per-category accuracy, with the closed-model advantage concentrated in the visual-grounding cluster.Table A6 compares all-model means and ranges with top-three closed and open ecosystem averages.
  • Limitations and use: The dataset is intended for academic research, not deployment-ready personality scoring, and inherits cultural and linguistic biases from predominantly Western-context English videos.The authors discourage personality screening or hiring use without consent, transparency, fairness audits, and attention to subjective cross-cultural interpretation.
  • Per-trait difficulty: Neuroticism is the hardest Big Five trait, averaging 37.7% accuracy and MAE 0.87 across the 27 evaluated models.Even the strongest models reach only 50–58% on Neuroticism, consistent with difficulty inferring internal emotional states from short clips.
  • Reasoning quality: Evidence Coverage is the hardest AI-as-Judge dimension, with mean 5.14, while Logical Coherence and Grounding Accuracy are easier.The pattern indicates that models can produce coherent explanations while under-citing specific behavioral evidence.

Q Effect of Reasoning Capability (Observational)

The reasoning-capability comparison is observational: reasoning-capable models lead most strongly on grounding-related measures, while rating differences are smaller, but confounding prevents causal attribution. Complementary visualizations show that trustworthy-grounding metrics reorder models and expose divergent cue retrieval despite similar ratings and explanations.

  • Caveat: The observed subset gap is not a controlled causal effect of explicit reasoning capability.Reasoning-capable variants are typically newer, larger, and drawn from stronger families, so the comparison is descriptive.
  • Observed group differences: +18.3 pp on T3, +11.5 pp on HR, and +2.6 pp on T1 distinguish reasoning-capable from non-reasoning subsets.Mean RGM is also more negative for reasoning-capable models (−3.8 versus +3.5), but the groups differ in size, family, generation, and training data.
  • Per-model visualization: Only 5 of 27 models occupy the high-T1, low-PR Trustworthy zone, with Gemma-4-31B-it the only open-source model there.The remaining models cluster above the PR=51.3% reference line, indicating a field-wide rather than archetype-specific prejudice pattern.
  • Failure modes: The four failure-mode rates form distinct per-model fingerprints: HR concentrates near the top, PR and CR saturate lower-ranked models, and IR is broadly elevated.Figure A7 presents these rates as a heatmap sorted by HR.
  • Rank reordering: Replacing T1 ranking with HR substantially reorders a small group of models, separating Confident Raters from Cautious Reasoners.Llama-4-Maverick-FP8 drops from rank 4 to 17, while Gemini 2.5 Flash rises from rank 24 to 5.
  • Sample-level divergence: On one video, GPT-4o and Gemini 3 Flash produce the same correct Extraversion rating and comparable T2 scores, but only Gemini answers the cue-grounding probes correctly.The example illustrates why matching ratings and plausible explanations can still lack retrievable behavioral support.

V Closed-vs-Open Frontier-Mean Task Gap

Closed-model frontier means lead open models on every task, with the largest ecosystem gap appearing in cue retrieval rather than rating or verbal reasoning. Across generations, T3 improves more than T1, while positional bias and model-specific T3 performance help explain grounding differences.

  • Frontier-mean comparison: Closed models lead open models on every task, and the ecosystem gap is several times larger on T3 than on T1 or T2.Together with the PR distribution, this places the closed-versus-open difference primarily in cue retrieval.
  • Generation trends: Within closed-model families, successive generations improve T3 substantially while T1 saturates earlier.GPT-4o to GPT-5.5 increases T3 from 31.9% to 66.4% (+34.5 pp), while Gemini 2.5 Pro to 3.1 Pro rises from 65.2% to 70.6%.
  • Positional bias: Every model with positional-bias σ >10 ranks in the bottom third on T3, and σ correlates negatively with T3 accuracy at r≈−0.68.The measure is proposed as a cheap early-warning signal for cue-retrieval collapse.
  • Model agreement: Top-10 models agree more on which videos are easy or hard for T1 than on which T3 questions they answer correctly.Mean off-diagonal Spearman correlation is 0.56 for T1 versus 0.39 for T3, making T3 more discriminative of model-specific competence.
  • Evaluation setup: The benchmark’s ecosystem comparisons use official APIs for proprietary models and H200 GPUs via vLLM for open-source models.Estimated compute resources for project phases are summarized in Table A11.
Loading 2605.22109v1…