PaperlayerView exact revision on arXiv

Source-linked AI summary

SVBench: Evaluation of Video Generation Models on Social Reasoning

Wenshuo Peng, Gongxuan Wang, Tianmeng Yang, Chuanhao Li, Xiaojie Xu, Hui He, Kaipeng Zhang

arXiv:2512.21507v5cs.CV

TL;DR

Video generators have improved perceptually but remain limited in socially coherent behavior, while benchmarks have not systematically assessed social reasoning in generated video. SVBench constructs a psychology-grounded benchmark and training-free agent pipeline, then evaluates current systems. Results show a gap between visual plausibility and deeper social reasoning, with leading proprietary systems ahead of open-source models in most dimensions.

  • Problem

    Video-generation benchmarks largely assess perceptual or physical qualities, and existing video social-reasoning benchmarks test interpretation rather than generation of coherent social interactions.

    Source: §1 · p. 2, §2.2 · p. 3

  • Method

    SVBench operationalizes psychological paradigms in a training-free agent pipeline that creates video prompts and evaluates generated clips across five social-reasoning dimensions.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Sora2-Pro and Veo-3.1 achieve overall pass rates of 79.6% and 72.4%, respectively, while open-source models perform substantially lower across nearly all dimensions.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The benchmark provides a theoretically grounded, interpretable basis for assessing whether video generators produce socially appropriate behavior beyond perceptual or physical plausibility.

    Source: §1 · p. 2, §5 · p. 8

    The primary benchmark uses fifteen short-video–feasible experiments; fifteen long-horizon experiments are reserved for an appendix because current generators typically produce only 5–10 second clips.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Recent text-to-video generation models have made remarkable progress in visual realism, motion fidelity, and text-video alignment, yet they still struggle to produce socially coherent behavior. Unlike humans, who readily infer intentions, beliefs, emotions, and social norms from brief visual cues, current models often generate literal scenes without capturing the underlying causal and psychological dynamics. To systematically assess this limitation, we introduce the first benchmark for social reasoning in video generation. Grounded in developmental and social psychology, the benchmark covers thirty classic social cognition paradigms spanning seven core dimensions: mental-state inference, goal-directed action, joint attention, social coordination, prosocial behavior, social norms, and multi-agent strategy. To operationalize these paradigms, we build a fully training-free agent-based pipeline that distills the reasoning structure of each paradigm, synthesizes diverse video-ready scenarios, enforces conceptual neutrality and difficulty control through cue-based critique, and evaluates generated videos with a high-capacity VLM judge along five interpretable dimensions of social reasoning. Using this framework, we conduct the first large-scale evaluation of seven state-of-the-art video generation systems. Results show a clear gap between surface-level plausibility and deeper social reasoning, suggesting that current models remain limited in their ability to generate socially grounded behavior. https://github.com/Gloria2tt/SVBench-Evaluation

1. Introduction

Current video generators can produce visually plausible scenes but often miss the intentions, beliefs, emotions, and norms that make behavior socially coherent. SVBench introduces a psychology-grounded benchmark and training-free pipeline to assess that gap.

1. Introduction

  • Existing video-generation benchmarks emphasize perceptual and physical plausibility, leaving socially appropriate behavior without an explicitly stated outcome under-assessed.The gap concerns whether generated behavior is socially and causally interpretable, not simply whether scenes look realistic.Source: §1 · p. 2
  • SVBench grounds its benchmark in seven social-cognition capacities, including mental-state inference, goal-directed action, joint attention, coordination, prosocial behavior, social norms, and multi-agent strategy.The taxonomy is drawn from developmental and social psychology.Source: §1 · p. 2
  • A four-agent, training-free pipeline distills psychological experiments, synthesizes diverse video-ready scenarios, critiques neutrality and difficulty, and evaluates videos with a vision–language model.The critic manipulates cues such as gaze, occlusion, and affordance visibility; the evaluator checks causal structure, social cues, and behavioral plausibility.Source: §1 · p. 2
  • The paper presents SVBench as the first dedicated benchmark and reports a systematic analysis of eight state-of-the-art video generators’ socially coherent behavior.The benchmark is designed to support large-scale assessment without human annotation.Source: §1 · p. 2

2. Related Work

Prior video-generation benchmarks mainly assess perceptual, physical, or action-level qualities, while existing video social-reasoning benchmarks assess interpretation of human-created clips. SVBench targets the untested task of generating socially coherent interactions.

2.1. Evaluation Benchmarks for Video Generation

  • Video-generation benchmarks cover visual quality, temporal stability, physics, and action-level consistency, but leave socially appropriate, paradigm-consistent behavior under-explored.Examples include VBench, EvalCrafter, VBench-2.0, PhyCoBench, and Morpheus.Source: §2.1 · p. 3

2.2. Social Reasoning in AI System

  • Video social-intelligence benchmarks assess recognition or question answering on existing human-created videos, not whether generators can synthesize coherent multi-agent interactions.The stated gap is generation of plausible beliefs, responses to others’ mental states, and norm-following behavior.Source: §2.2 · p. 3

3. Method

The method builds a psychology-grounded seed suite, converts feasible experiments into neutral, difficulty-controlled video prompts, and scores generated clips with a structured VLM judge. The design separates social-reasoning assessment from visual plausibility while accommodating current short-video limits.

3.1. Seed Suite and Feasible Subset

  • Thirty experiments span seven social-cognition dimensions, with fifteen short-video–feasible experiments forming the primary benchmark and fifteen long-horizon tasks reserved for an appendix.Long-horizon tasks require multiple events or extended sequences and are considered unsuitable for current short-form generators.Source: §3.1 · p. 3, Table 1 · p. 5

3.2. Agent-Based Generation

  • The Experiment Understanding Agent decomposes each paradigm into its psychological phenomenon, key concepts, test point, and expected outcome before prompt generation.This intermediate specification is intended to preserve theoretical fidelity and reduce conceptual drift.Source: §3.2 · p. 4
  • The Prompt Synthesis Agent turns abstract constructs into observable, concrete actions in video-ready scenarios while withholding mental states and expected outcomes.Scenarios are designed for 5–10 second clips and keep action descriptions distinct from evaluation targets.Source: §3.2 · p. 4
  • The Critic Agent removes interpretive language and ground-truth leakage, then adjusts psychological, motoric, and contextual cues to create difficulty variants.Hard variants remove or obscure central cues; failed checks trigger diagnostic feedback and iterative prompt revision.Source: §3.2 · pp. 4–5

3.3. Agent-Based Evaluation

  • Because social scenarios can have many valid realizations, evaluation tests whether a paradigm’s intended social logic appears rather than matching a single reference video.The VLM judge uses five binary dimensions instead of continuous scores to reduce calibration noise and inter-trial variance.Source: §3.3 · p. 5
  • The five dimensions assess paradigm replication, prompt faithfulness, social coherence, social-cue effectiveness, and video plausibility, with each scored 0 or 1.The overall score averages the five assessments; video plausibility provides a baseline for visual stability, helping distinguish generation failures from reasoning errors.Source: §3.3 · p. 5
  • The evaluation design is intended to disentangle generation and reasoning failures, reduce VLM calibration noise, and scale to more complex social scenarios.Its structured dimensions support interpretable diagnosis of failure modes.Source: §3.3 · p. 5

4. Experiment

The experiments evaluate short-form social reasoning tasks across video-generation systems, then assess prompt quality, cue-based difficulty ordering, and alignment between automated and human evaluation. Proprietary systems lead overall, while the validation analyses show how performance and evaluation vary across reasoning demands.

4. Experiment

  • The main experiment evaluates 15 short-form tasks with three difficulty levels and three prompts per level, totaling 135 prompts; long-form tasks are reported in the supplement.The tasks were selected to fit within a single 5–10 second video segment.Source: §4 · p. 5

4.2. Evaluation Results

  • Sora2-Pro and Veo-3.1 lead nearly all categories with 79.6% and 72.4% overall pass rates, while open-source systems trail across most dimensions, with Wan2.2 strongest.Hailuo02-S and Kling2.5-Turbo score 56.4% and 52.2% overall, respectively.Source: §4.2 · p. 6

4.3. Verification of the Agent-Based Generation

  • Pass rates rise from roughly 67.7% without conceptual understanding to over 75.8% with the test point distilled and above 86.7% after Critic Agent refinement.Human judges assessed whether prompts expressed the intended construct while remaining neutral and free of answer leakage.Source: Table 3 · p. 6, §4.3 · p. 7
  • Cue variants yield Easy > Medium > Hard ordering for Hailuo02-S and Kling2.5-Turbo, but Sora2-Pro and Veo3.1 peak at medium or hard conditions.The authors attribute the difference to stronger intrinsic reasoning in the latter models and possible conflicting or redundant cues in short clips.Source: §4.3 · p. 7

4.4. Validation of the Agent-Based Evaluation

  • VLM-judge and human-score profiles show closely matched trends, but humans are more lenient on perceptual dimensions and stricter on reasoning-intensive dimensions.Human annotators evaluated a stratified sample of 160 clips using the same five-dimensional rubric.Source: Figure 3 · p. 7, §4.4 · pp. 7–8
  • Qualitative cases contrast a plausible but incorrect gaze-following interaction, scored zero on four dimensions, with successful instrumental helping scored fully across all five.The first video showed direct conversation rather than the prompted gaze-following behavior; the second depicted a boy helping a woman reach a pen.Source: §4.4 · p. 8

5. Conclusion

SVBench provides a psychology-grounded benchmark and training-free pipeline for evaluating social reasoning in generated videos. Across eight video generators, the authors report a gap between perceptual realism and social coherence, with systematic failures on several social reasoning capacities.

  • The benchmark operationalizes eight components of social cognition through thirty classic psychological experiments.Source: §5 · p. 8
  • A four-agent pipeline converts psychological paradigms into validated, difficulty-controlled video prompts and evaluates outputs with a vision–language model judge.The authors report that generation-side and evaluation-side agents align strongly with human reasoning, enabling assessment without manual annotation.Source: §5 · p. 8
  • Across eight state-of-the-art video generators, leading proprietary systems show emerging competence in goal understanding, joint attention, and prosocial behavior, but even the strongest fail systematically on belief inference, subtle cue integration, and multi-agent coordination.Source: §5 · p. 8
Loading 2512.21507v5…