PaperlayerView exact revision on arXiv

Source-linked AI summary

SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos

Zheyu Huang, Zijing Shi, Haozhe Luo, Huadong Tang, Mingyu Liu, Meng Fang, Ling Chen

arXiv:2608.30716v1cs.CLcs.CV

TL;DR

Existing video benchmarks provide limited evidence about whether LMMs reason over social dynamics beyond a single observed trajectory. SocialReasonBench addresses this gap with a branching-narrative benchmark and game-grounded curation pipeline, finding that models handle basic social understanding better than causal and counterfactual reasoning.

  • Problem

    Existing social video benchmarks usually present fixed trajectories, making genuine social reasoning difficult to distinguish from exploitation of recurring narrative patterns.

  • Method

    SocialReasonBench uses interactive-narrative gameplay, seven social-reasoning dimensions, and a multi-agent pipeline that grounds questions and labels in checkable game-state signals.

  • Results

    Current LMMs perform relatively well on basic social understanding but struggle with causal antecedent inference and counterfactual reasoning, often relying on incomplete modality cues or salient visual shortcuts.

  • Takeaways & Limitations

    The results indicate a gap between observable event recognition and deeper reasoning over latent social states and alternative narrative outcomes.

  • Takeaways & Limitations

    The benchmark is built from a single English-language game and may reflect that game's cultural, stylistic, and narrative characteristics.

Abstract

from arXiv · show

Recent advances in Large Multimodal Models (LMMs) have greatly improved video understanding, yet their ability to reason about human-centered social situations remains limited. Existing benchmarks typically rely on videos with a single observed trajectory, making it difficult to determine whether models truly understand social dynamics or merely exploit recurring narrative patterns. We introduce SocialReasonBench, a video multiple-choice QA benchmark for evaluating socially grounded reasoning in scenarios derived from interactive narratives. Built from gameplay videos of Detroit: Become Human, the benchmark leverages branching storylines where player decisions lead to alternative social outcomes that can be checked against the game's own script, flowchart, and recorded branches. We develop a multi-agent curation pipeline that localizes socially meaningful clips, grounds answer labels in game-state signals, and generates theory-guided questions with diagnostic distractors. SocialReasonBench covers seven reasoning dimensions, including intent recognition, emotional empathy, moral dilemma, counterfactual reasoning, and causal antecedent. Experiments on contemporary LMMs show that models perform reasonably well on basic social understanding but struggle with counterfactual and causal reasoning. Further ablation and diagnostic error analyses reveal that models often depend on incomplete modality cues and fall into reasoning traps such as visual shortcuts, highlighting a gap between observable event recognition and deeper reasoning over latent social states.

1 Introduction

SocialReasonBench addresses limits in social video reasoning evaluation by using branching interactive narratives with verifiable game-state outcomes. It evaluates fine-grained social reasoning and finds that LMMs handle basic understanding better than causal and counterfactual reasoning.

  • Existing social video benchmarks largely use fixed, linear narratives and post-hoc annotation, limiting tests of genuine reasoning and introducing potential interpretive bias.
  • SocialReasonBench uses branching gameplay narratives to make alternative outcomes observable and construct naturally paired counterfactual scenarios.
  • The benchmark assesses seven fine-grained dimensions, including intent recognition, moral dilemma, counterfactual reasoning, and causal antecedent.
  • Its multi-agent curation pipeline localizes socially salient clips, generates theory-informed questions, and grounds answer labels in verifiable game-state signals.
  • SocialReasonBench contains over 500 video-QA pairs, each with a short clip, one question, six candidate answers, and one ground-truth label.
  • Across evaluations of eight LMMs, models perform better on basic social understanding but struggle with causal antecedent and counterfactual reasoning.

2 Related Work

Related work spans general video reasoning, multimodal social reasoning, and counterfactual evaluation. SocialReasonBench extends these directions toward socially grounded reasoning over alternative narrative outcomes.

  • Recent video LMMs process temporally structured visual inputs through representation learning, multimodal alignment, and instruction tuning.
  • Video benchmarks such as MVBench, Video-MME, EgoSchema, and LongVideoBench evaluate broad video understanding capabilities.
  • Multimodal social reasoning benchmarks study implicit social states through narrative understanding and commonsense inference over events, intents, and consequences.
  • Counterfactual reasoning evaluates whether models can predict how outcomes change when a different action or event occurs.

3 SocialReasonBench

SocialReasonBench is built from branching interactive-game narratives to evaluate socially grounded video reasoning with game-verifiable answers. Its multi-agent pipeline selects meaningful clips, grounds outcomes in game signals, and generates theory-guided questions spanning interaction types and reasoning abilities.

  • Benchmark construction: SocialReasonBench uses Detroit: Become Human’s branching narrative structure to expose alternative socially situated outcomes for benchmark questions.Chapter flowcharts represent narrative states, possible actions, and deterministic transitions, with unrealized outgoing edges defining counterfactual branches.
  • Benchmark construction: The benchmark formulates each instance as a video multiple-choice question with anonymized background, candidate options, and a game-grounded correct label.Labels are determined from hidden game signals, while generated distractors remain plausible but inconsistent with the grounded branch logic.
  • Multi-agent synthesis: A Director Agent selects socially meaningful clips and maps them to interaction and reasoning categories, while a Tracker Agent aligns clips with states and hidden game signals.The Generator Agent then synthesizes theory-guided questions, correct answers, and diagnostic distractors from the selected clip and grounded signals.
  • Multi-agent synthesis: Entity anonymization replaces game-specific names and concepts with neutral placeholders to reduce reliance on memorized walkthroughs and game-specific priors.Examples include replacing character, organization, and android-model identifiers with semantically neutral alternatives.
  • QA curation criteria: The taxonomy combines four social interaction types with seven reasoning capabilities, including intent recognition, empathy, moral dilemma, counterfactual reasoning, and causal antecedent.Each instance receives a coordinate in the two-axis taxonomy, which guides clip selection and question generation.
  • QA curation criteria: 56.4% of labels are directly verifiable from in-clip UI evidence or the official flowchart, while 43.6% require a short derivation from those documented sources.Moral-dilemma labels are defined using large-scale human player choice statistics rather than normative judgments.

4 Experiments

Experiments evaluate eight closed- and open-weight LMMs using accuracy and Trap Fall Rate, revealing stronger performance on basic social understanding than on causal and counterfactual reasoning. Audio ablation and diagnostic analyses show that models need acoustic evidence yet often follow visual, knowledge-based, or logic traps.

  • Experimental Setup: Eight representative closed- and open-weight LMMs are evaluated on SocialReasonBench in a zero-shot setting.The evaluated systems include five closed-weight and three open-weight models.
  • Metrics: Accuracy measures overall task performance, while Trap Fall Rate measures the proportion of incorrect predictions assigned to each diagnostic trap type.Together, the metrics capture both whether models fail and what misleading evidence they follow.
  • Main Results: Models generally perform better on intent recognition, behavior prediction, and emotional empathy than on reasoning over latent states, causal dependencies, or alternative outcomes.The comparison is emphasized within each model because input regimes differ across systems.
  • Main Results: 85.14% accuracy on GPT-4o’s intent recognition falls to 50.00% on counterfactual reasoning and 28.57% on moral dilemma questions.The moral-dilemma subset contains only 21 instances, so its per-model values are treated as indicative rather than conclusive.
  • Main Results: Human evaluators achieve approximately 85%–93% accuracy across all seven reasoning dimensions, whereas LMMs show larger drops on causal and counterfactual reasoning.For humans, the reported decrease from 90.8% to 86.9% is not statistically significant.
  • Modality Ablation Analysis: Removing audio from Gemini 3 Pro reduces overall accuracy by approximately 6%, with the largest declines in Causal Antecedent and Emotional Empathy.The muted model is more likely to rely on surface visual cues when spoken content, tone, or acoustic context supplies critical evidence.
  • Diagnostic Trap Analysis: Diagnostic trap profiles differ by model: stronger closed-weight systems often follow knowledge or visual traps, while the smaller open-weight model more often follows logic traps.These patterns correspond respectively to external priors or salient cues versus difficulty tracking temporal dependencies and branching state transitions.
  • Diagnostic Case Study: In a branching helicopter scenario, GPT-4o selects a real-world SWAT distractor instead of the game-specific mechanism linking helicopter presence to psychological pressure.For causal antecedent questions, models also attribute Beta’s breakdown to personality rather than Alpha’s refusal to withdraw the helicopter.

5 Conclusion

SocialReasonBench evaluates social reasoning in LMMs using interactive narrative games and game-state-grounded questions. Current models handle basic social understanding relatively well but struggle with causal antecedent inference and counterfactual reasoning.

  • SocialReasonBench is a video-QA benchmark built on interactive narrative games and a multi-agent curation pipeline.
  • Its questions are grounded in game-state signals that can be checked against the game’s own record.
  • Current LMMs perform relatively well on basic social understanding but struggle with causal antecedent inference and counterfactual reasoning.
  • Ablation and error analyses show that plausible but incorrect cues can mislead models when answers depend on branch-specific causal evidence.
  • The findings point to a need for multimodal models that better track latent social states and alternative narrative outcomes.

Limitations

The benchmark is derived from one English-language interactive narrative game, so its scenarios reflect that game’s cultural, stylistic, and narrative characteristics. Its android protagonists also make the social reasoning setting one step removed from reasoning about humans, while future work can broaden the source coverage.

  • SocialReasonBench is built from a single English-language interactive narrative game.This source provides dense branching trajectories, rich social interactions, and verifiable game-state signals.
  • The benchmark naturally reflects the cultural, stylistic, and narrative characteristics of its particular source game.
  • Because its protagonists are androids, reasoning about their interactions remains one step removed from reasoning about humans.
  • Future work can extend the construction pipeline to additional narrative-driven games, broadening social scenarios and interaction contexts.
  • The curation pipeline uses Gemini-family models also included in evaluation, creating a possible generator-family style advantage.Excluding both Gemini systems leaves the main finding unchanged; future releases could diversify curation models or add human-written subsets.

A Copyright and Data Release

The benchmark uses publicly available gameplay videos and community resources under a stated non-commercial release policy. The data process preserves rights-holder ownership, limits redistribution, and includes diagnostic curation procedures and anonymization safeguards.

  • The benchmark uses publicly available YouTube gameplay videos of Detroit: Become Human without modifying the game or extracting proprietary assets.
  • Community resources such as dialogue transcripts and walkthroughs assist clip segmentation but are not redistributed.
  • The benchmark is released for non-commercial research use with short videos, derived metadata, and original annotations under CC BY-NC-SA 4.0.
  • Game-related audio-visual assets remain property of their rights holders, and users are responsible for copyright, platform, and licensing compliance.
  • Diagnostic distractors are assigned through fixed internal option IDs, while their final presentation order is randomized during evaluation.
  • Model-facing text is anonymized to reduce franchise-prior contamination, while backend traces retain real names for audit and reproducibility.

D Example Data Unit

The example data-unit and processing details show how SocialReasonBench structures model inputs, tracks game-state signals, and handles branching, multi-character gameplay. Backend grounding traces are stripped from evaluated inputs, while automated processing is followed by deduplication and human verification.

  • Example Data Unit: Evaluated models see only the model_input block; backend pipeline context, grounding traces, and diagnostic analyses are stripped beforehand.
  • Example Data Unit: Across 40 source script files, the pipeline produces 395 nodes, 679 candidate coordinates, and 730 candidate clips before deduplication and human verification.
  • Example Data Unit: The pipeline tracks seven categories of in-game social signals, including relationship changes, instability shifts, and probability-of-success deltas.
  • Example Data Unit: Causal locks appear 96 times overall and serve as the primary scaffold for Causal Antecedent questions.
  • Example Data Unit: The complete Data Unit JSON provides deterministic evaluation while isolating evaluated models from backend grounding traces.
  • Branching and Multi-Character Processing: For multi-character videos, protagonist-based timeline partitioning prevents nodes from being routed to another character’s gameplay segments.
  • Branching and Multi-Character Processing: Merging adjacent same-character segments reduces processing from one call per segment to one call per character.For Crossroads Part 1, this reduces 30 calls to 3 calls.
  • Branching and Multi-Character Processing: The three multi-character videos contribute 128 clips in total: 43 from Crossroads Part 1, 55 from Crossroads Part 2, and 30 from Night of the Soul.

H Evaluation Hyperparameters

The evaluation uses model-specific input configurations and deterministic, structured-output settings, while acknowledging that modality support complicates architectural comparisons.

  • Pipeline-side Models: Curation agents use Gemini 3.1 Pro Preview at temperature 0.1–0.2, while the Generator uses Gemini 2.5 Pro at temperature 0.3.Higher synthesis temperatures support controlled stochasticity; evaluation temperatures are deterministic.
  • Input Configurations: Native-video models receive raw video, whereas keyframe-based models receive 32 evenly sampled frames from each precise clip.Input modality support is part of the performance difference and should not be treated as purely architectural.
  • Output and Reliability Controls: Both curation and evaluation request structured JSON, using schema-bound decoding where supported and tolerant regex parsing for Anthropic and GLM.All LLM calls use exponential backoff for HTTP 429 responses, with up to four attempts.

I Human Evaluation Protocol

Human evaluation uses a blind interface, balanced single-annotation coverage, and an overlap-based agreement check, while carefully distinguishing its reference role from a human ceiling.

  • Sampling Design: Five author-team annotators each completed 107 questions, covering all 532 benchmark instances at least once.Three duplicated instances supported a consistency sanity check but were excluded from headline HE accuracy.
  • Annotation Interface: Annotators saw the clip with original audio, anonymized background, question, and six permuted options, while backend grounding and correct labels remained hidden.They were instructed to watch the full clip, use only the clip and background, and avoid external walkthroughs.
  • Aggregation: Headline HE accuracy uses one primary annotation per instance and is computed as a per-dimension average against the benchmark reference label.Skipped questions are excluded from the denominator.
  • Inter-Annotator Agreement: The 50-item stratified overlap set was independently re-annotated by all five annotators to estimate agreement under the same blind interface.Agreement is computed over shown option letters, while the primary pass remains largely disjoint.
  • Grounding Safeguards: Human labels are grounded through source-verifiability tiers, including explicit UI signals and documented flowchart or script evidence.Tier C instances undergo source tagging, human verification, and a unique branch-precondition check; speculative or non-unique cases are excluded.

J.3 Per-Dimension Breakdown

More inferential reasoning dimensions have stronger external grounding, so their performance deficits are less likely to arise solely from labeling artifacts.

  • Per-Dimension Grounding: 58%–90% of the more inferential dimensions have Tier A, B, or AB grounding, according to the per-dimension tier distribution.These dimensions are also where contemporary LMMs show larger performance drops.
  • Diagnostic Interpretation: Causal antecedent and other difficult inferential dimensions are not the dimensions with the weakest external grounding.This pattern supports the benchmark’s diagnostic value.
  • Safeguards: Tier C risk is controlled through source tagging, human rejection of speculative labels, and a unique branch-precondition requirement.Instances relying solely on unconstrained multi-step Tracker inference are excluded.
  • Moral Dilemma: Moral Dilemma references use the option selected by the largest recorded player share in the game’s World’s Stats feature.This reference is a human-majority label rather than a deterministic utility judgment.

K Statistical Reliability and Robustness

Statistical reporting quantifies uncertainty and limits human–model comparisons, while robustness checks show that the central performance pattern persists without Gemini systems.

  • Statistical Reliability: Wilson 95% confidence intervals quantify binomial uncertainty for every accuracy entry in Table 3.The intervals are computed from integer correct counts and can be reconstructed from the table.
  • Comparison Scope: Human–model comparisons are descriptive because the two evaluations use different input regimes.This limits interpretation of direct human-versus-model accuracy differences.
  • Robustness Check: Five of six remaining models still show a significant perception-versus-causal/counterfactual decrease after Holm correction when both Gemini systems are excluded.ARC-Hunyuan-Video is the only exception, so the qualitative finding does not depend on the Gemini family.
  • Evaluation Controls: All evaluated systems receive the same fixed closed-world prompt template, with deterministic option permutations across all 532 instances.The protocol requires reasoning from the video and stated background rather than external franchise priors.
Loading 2608.30716v1…