Source-linked AI summary
GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions
Junho Kim, Xu Cao, Houze Yang, Bikram Boote, Ana Jojic, Fiona Ryan, Bolin Lai, Sangmin Lee, James M. Rehg
TL;DR
MLLMs struggle to interpret who interacts with whom from gaze and gesture in multi-person videos. GRASP connects grounded social reasoning data with Social Grounding Reward, improving GRASP-Bench performance while preserving zero-shot performance on related tasks.
Problem
Existing research largely treats gaze and gesture as isolated perceptual cues rather than interpreting their social meaning in context.
Method
GRASP constructs identity-anchored social events from gaze trajectories and deictic gestures, while Social Grounding Reward verifies participant-level grounding in model reasoning.
Results
Participant-grounded supervision improves performance on GRASP-Bench and preserves zero-shot performance across related social video QA benchmarks.
Takeaways & Limitations
GRASP provides a unified resource for training and evaluating grounded social reasoning over multi-person gaze and gesture interactions.
Takeaways & Limitations
The annotations can remain noisy in ambiguous scenes, and SGR mainly verifies participant-level grounding without stricter temporal alignment or richer social cues.
Abstract
from arXiv · showhide
Understanding social interactions requires reasoning over subtle non-verbal cues, yet current multimodal large language models (MLLMs) often fail to identify who interacts with whom in multi-person videos. We introduce GRASP, a large-scale social reasoning dataset that connects high-level social QA with fine-grained gaze and deictic gesture events. GRASP contains 290K question--answer pairs over 46K videos totaling 749 hours, organized by a 16-category taxonomy spanning gaze, gesture, and joint gaze--gesture reasoning, together with GRASP-Bench for evaluation. Unlike prior resources that focus on either isolated cues or high-level social QA, GRASP builds questions from identity-consistent gaze trajectories, deictic gestures, and their joint compositions into social events. Moreover, we propose Social Grounding Reward (SGR), a learning signal that uses these social events to encourage models to reason about the participants involved in each interaction. Experiments show that SGR improves performance on GRASP-Bench while maintaining zero-shot performance on related social video QA benchmarks.
1 Introduction
GRASP addresses the gap between perceiving isolated gaze and gesture cues and reasoning about their social meaning among specific participants in multi-person videos. It combines grounded social data with Social Grounding Reward to train MLLMs to base reasoning on relevant non-verbal evidence and transfer this ability across benchmarks.
- Motivation and Problem: Prior methods identify gaze targets or gesture categories as perceptual primitives, but do not systematically reason about who interacts with whom or what the cues mean socially.This gap motivates grounded reasoning over participant relationships rather than cue presence alone.
- GRASP Dataset and Benchmark: GRASP jointly supports high-level social reasoning and fine-grained gaze–gesture grounding in multi-person videos, addressing fragmented supervision in existing social datasets.The dataset and benchmark are built for training and evaluating grounded social reasoning.
- Results: Post-training with GRASP data and policy achieves strong GRASP-Bench performance and generalizes to three other social reasoning benchmarks.These results indicate that social grounding transfers across task formulations and helps MLLMs move beyond primitive perception.
- GRASP Dataset and Benchmark: GRASP-Bench evaluates whether MLLMs can reason over participant-level gaze and gesture interactions instead of relying on scene-level shortcuts.The framework uses a bottom-up pipeline to scale participant-level social event annotation.
- Social Grounding Reward: Social Grounding Reward directly verifies whether reasoning attends to the correct participants involved in the underlying social events.It supplies an RL signal that ties generated reasoning to relevant non-verbal cues, unlike policy optimization that does not explicitly make this connection.
2 Related Works
Prior work has developed gaze understanding and gesture recognition as largely parallel research tracks, while recent MLLMs have advanced video understanding. However, jointly modeling gaze and gesture remains an identified gap in social AI.
- Gaze and Gesture in Human Social Behavior: Gaze understanding has focused on gaze target estimation using benchmarks such as GazeFollow and VideoAttentionTarget.These benchmarks support research on interpreting gaze as a cue to social intention and utterance targets.
- Gaze and Gesture in Human Social Behavior: Gesture recognition has been studied across RGB-D, high-resolution image, egocentric, and multi-person social-context datasets.Examples include LD-ConGR, HaGRID, EgoGesture, and SocialGesture, alongside modeling with MLLMs and VLMs.
- Gaze and Gesture in Human Social Behavior: Despite progress in gaze and gesture research, jointly modeling the two non-verbal cues remains a critical gap.The supplied related-work discussion contrasts advances in individual domains with the unresolved need for joint modeling.
- Multi-modal Large Language Models in Social AI: Recent MLLMs have demonstrated video understanding across multimedia analysis and human action recognition, using extensive context windows for complex video inputs.The passage identifies Gemini 3, GPT-5, and Qwen-VL as proprietary and open-source examples, respectively.
3 GRASP Dataset: Grounded Reasoning And Social Perception
GRASP constructs a large-scale dataset by grounding multi-person social reasoning in identity-consistent gaze trajectories, deictic gestures, and unified social events. Its 290K QA pairs follow a 16-category taxonomy spanning gaze, gesture, and joint gaze–gesture reasoning.
- Dataset construction: GRASP comprises 46K videos and represents interactions with person-identity-anchored gaze trajectories and deictic gestures integrated into higher-order social events.These structured events provide supervision for QA generation and the grounding reward.
- Gaze annotation: Five gaze-event types yield 169K events: mutual gaze, joint attention, gaze following, attention capture, and sudden gaze shift.The events are grounded in established gaze-behavior taxonomies.
- Gesture annotation: The gesture pipeline detects pointing, showing, giving, and reaching, then self-refines predictions to produce 76K reliable person-level gesture instances.Each instance includes initiator and target identities before gaze and gesture events are merged into a unified social graph.
- QA taxonomy: The 16-category taxonomy covers six gaze tasks, six gesture tasks, and four joint gaze–gesture tasks with increasing reasoning difficulty.Joint categories include temporal alignment, gaze response to gesture, eye contact during interaction, and cross-modal person dynamics.
- QA generation: 290K final QA pairs are generated from structured event metadata using participant identities, temporal intervals, and interaction types, followed by assessment-based refinement and filtering.Each QA is anchored around a target event and augmented with a short temporal window.
4 Learning to Ground Social Reasoning
GRASP trains models in two stages: structured supervised warm-up followed by reinforcement post-training with correctness, formatting, structural grounding, and social grounding rewards. The grounding rewards make reasoning explicitly represent gaze and gesture evidence while identifying the participants involved in each interaction.
- Training Strategy: The two-stage strategy first warms up models with structured SFT, then applies RL using correctness, format consistency, structural grounding, and social grounding rewards.This staged design provides a supervised starting point before policy optimization.
- Stage 1: Structured Format Supervised Fine-Tuning: Direct RL suffers from cold start because existing multimodal corpora rarely support fine-grained gaze and gesture reasoning, producing sparse relevant rollouts and rewards.The warm-up addresses this sparse-reward bootstrapping problem.
- Stage 1: Structured Format Supervised Fine-Tuning: Structured SFT teaches models to output separate <gaze> and <gesture> evidence before an answer, producing a reference model for subsequent RL.The format enumerates temporally grounded gaze and gesture events from the social graph.
- Stage 2: Grounded Reinforcement Post-Training: Structural grounding rewards explicit articulation of gaze and gesture observations through dual tags rather than leaving non-verbal cues implicit.This encourages a consistent reasoning-trace structure.
- Stage 2: Grounded Reinforcement Post-Training: Social grounding compares predicted participant identities with target participants using weighted precision and recall, penalizing irrelevant mentions and missed key individuals.The reward targets relational reasoning over the people involved in the social event.
5 Experiments
Experiments evaluate GRASP-Bench across 16 categories and test zero-shot generalization on three social-reasoning benchmarks. Social Grounding Reward consistently improves grounded social reasoning, with gains over supervised and generic RL baselines, stronger scaling, and a positive link between participant precision and accuracy.
- Experimental Setting: Evaluation covers 16 GRASP-Bench categories plus zero-shot testing on MMSI, Online-MMSI, and TVQA+.MMSI and Online-MMSI include speaker target identification, pronoun coreference resolution, and mentioned-player prediction.
- Main Results: Socially-grounded RL consistently outperforms supervised fine-tuned and existing RL post-training baselines across reasoning types, especially gaze, gesture, and cross-modal reasoning.The results support explicitly grounding reasoning in nonverbal evidence rather than relying on coarse visual perception or language priors.
- Cross-Benchmark Generalization: SGR-trained models outperform or remain competitive with corresponding instruction-tuned and RL post-training baselines without task-specific fine-tuning on the three external benchmarks.Post-training is performed only on GRASP, indicating no over-specialization to the GRASP-Bench format.
- Comparison with GRPO: Adding SGR produces substantially larger gains across all metrics than adding GRPO alone, showing that explicit grounding contributes beyond reasoning supervision.GRPO yields moderate improvements, while ungrounded reasoning traces often remain verbose and fail to identify relevant individuals.
- Scaling with Model Size: Average accuracy improves with model size, increasing from 46.2 to 50.4 and from 45.3 to 52.6 across the two baseline backbones.Scaled SGR-trained models remain in the high grounded-participant-precision region while achieving stronger GRASP-Bench accuracy.
- Grounding Precision Analysis: Grounded-participant precision positively correlates with overall accuracy, with ρ = 0.64 across reasoning baselines.The analysis measures the proportion of correctly mentioned participants in each reasoning trace alongside accuracy.
6 Discussion and Conclusion … C Additional Analyses and Experimental Details
GRASP connects high-level social QA with fine-grained gaze and gesture events, while SGR encourages participant-level grounding. The paper also documents construction procedures, dataset statistics, additional experimental details, and limitations including noisy annotations and incomplete temporal or social-cue alignment.
- 6 Discussion and Conclusion: SGR mainly verifies participant-level grounding, while stricter temporal alignment and richer social cues remain important future work.Broader limitations are discussed in Appendix E.
- 6 Discussion and Conclusion: GRASP provides 290K QA pairs over 46K videos, organized by a 16-category taxonomy and accompanied by GRASP-Bench for social reasoning evaluation.The dataset connects high-level social QA with fine-grained gaze and gesture events in multi-person videos.
- 6 Discussion and Conclusion: SGR uses structured social events to encourage models to reason about the participants involved in each interaction.The discussion characterizes SGR as a simple but effective reward signal for participant-level grounding.
- A Detailed GRASP Construction: Appendix A also includes quality control and human validation as part of the detailed construction process.The discussion notes that annotations can remain noisy in ambiguous scenes despite quality control and human validation.
- B Detailed GRASP Data Statistics: Appendix B reports per-dataset scale, gaze and gesture distributions, QA categories, construction yield, event density, training distribution, and comparisons with specialized social VQA benchmarks.These analyses are listed in Sections B.1–B.6.
- C Additional Analyses and Experimental Details: Appendix C includes training details, other benchmark details, participant-ID corruption analysis, reasoning-length breakdowns, and an SGR error profile.These additional analyses and experimental details are listed in Sections C.1–C.5.
D Qualitative Comparison … A.6 Social Reasoning QA Generation
The supplied material outlines GRASP’s detailed construction pipeline, from identity-consistent gaze and gesture extraction through unified social graphs and validated reasoning QA generation. It also lists qualitative-comparison subsections, but provides no substantive qualitative findings.
- D Qualitative Comparison: The qualitative-comparison material lists GRASP-Bench examples, baseline reasoning-trace comparisons, and failure cases without providing their contents.
- A.1 Video Pre-processing and Person-Consistent Gaze Trajectories: Person tracking with SAM 3 and face detection produce temporally consistent participant identities and localized facial regions for gaze estimation.
- A.1 Video Pre-processing and Person-Consistent Gaze Trajectories: The final per-frame representation aligns each participant with person and face boxes, a normalized gaze point, and an in-frame validity flag.
- A.2 Gaze Feature Extraction: Gaze features use conservative missing-data interpolation and face-centered velocity, improving robustness to camera pans and global frame motion.
- A.3 Social Gaze Event Detection: Five social gaze events are detected from identity-consistent trajectories: sudden gaze shift, joint attention, gaze following, attention capture, and mutual gaze.
- A.4 Deictic Gesture Annotation: Deictic gesture annotation identifies pointing, showing, giving, and reaching using explicitly identity- and time-labeled videos under conservative visual criteria.
- A.5 Unified Social Graph Construction: Gaze and gesture events become temporally ordered social-graph nodes containing participant identities, timing, confidence, and modality-specific attributes.
- A.6 Social Reasoning QA Generation: QA generation converts graph events into verifiable questions across 16 gaze, gesture, and joint-reasoning categories, with adaptive difficulty and structured-field validation.
A.7 Quality Control and Human Validation
GRASP applies layered automated quality control to filter uncertain annotations and malformed events while preserving structures needed for joint reasoning. An independent human audit finds substantial benchmark validity, with gesture and joint reasoning outperforming overall validity while gaze reasoning remains more difficult to verify.
- Automated Quality Control: Confidence filtering, conservative gesture annotation, and graph cleanup reduce annotation noise while preserving gaze–gesture pairs required for joint reasoning.The cleanup removes invalid temporal boundaries, duplicate events, and malformed event records.
- Automated Quality Control: A graph-level health check verifies that event type, participants, start time, end time, and confidence are present across the corpus.
- Human Validation: Four evaluators audit approximately 25% of GRASP-Bench in a category-balanced sample, judging whether each QA instance is visually supported, temporally grounded, and unambiguous.They inspect the corresponding video clip, participant IDs, temporal context, category label, question, and answer options.
- Human Validation: 74.2% overall validity is achieved, with gesture and joint reasoning reaching 78.9% and 80.0%, respectively.Gaze reasoning is comparatively more challenging because fine-grained gaze direction, temporal ordering, and group attention are difficult to verify in dense multi-person scenes.
B Detailed GRASP Data Statistics … B.4 Construction Yield
GRASP spans diverse multi-person social video sources and organizes identity-consistent gaze and gesture annotations into structured social representations. Its statistics cover dataset scale, event distributions, a 16-category QA taxonomy, and construction yield.
- B Detailed GRASP Data Statistics: GRASP draws from multi-party TV dialogue, social intelligence, multi-speaker conversation, embodied multi-person behavior, and social deduction gameplay, augmented with online videos.The online-video augmentation targets dense multi-person interactions with rich gaze and gesture signals.
- B Detailed GRASP Data Statistics: 46,158 processed videos yield identity-consistent gaze trajectories and deictic gestures integrated into temporally ordered social graphs.The pipeline detects 429,663 raw gaze events, retains 215,378 high-confidence gaze events, and retains 88,234 gesture events.
- B.1 Per-Dataset Scale: Social Deduction Game contains 4,044 videos and contributes 127,033 gaze events plus 30,006 gestures, reflecting dense multi-person interaction patterns.This dataset is highlighted as a particularly interaction-dense source in the per-dataset scale analysis.
- B.1 Per-Dataset Scale: The benchmark contains 16 QA categories: six gaze categories, six gesture categories, and four joint reasoning categories.The category distribution distinguishes modalities through bar colors and difficulty through markers.
- B.2 Gaze and Gesture Event Distributions: Retained gaze events include five types, with gaze following most frequent at 31.2%; pointing dominates gestures at 52.5%.Attention capture is the second-most frequent gaze category at 23.1%, while gesture annotations include pointing, reaching, showing, and giving.
- B.3 QA Category Distribution: The QA taxonomy groups gaze reasoning, gesture reasoning, and joint reasoning, covering target identification, temporal reasoning, reciprocal patterns, sequence chains, and cross-modal relations.Joint reasoning links gaze and gesture events over time.
- B.4 Construction Yield: The confidence filter retains 215,378 of 429,663 raw gaze detections, corresponding to 50.1% retention, before merging events into 181,926 timeline events.Timeline events support multiple question forms, including recognition, temporal comparison, sequence reasoning, and cross-modal reasoning.
B.5 Event Density and Training Distribution … C.2 Other Benchmark Details
GRASP combines dense, modality-balanced training data with a two-stage SFT-to-RL procedure and evaluates transfer to related human-centered video understanding benchmarks. Its analyses characterize source-domain event density, training-example difficulty, benchmark distinctions, and evaluation settings.
- B.5 Event Density and Training Distribution: Social Deduction Game has the highest event density, averaging 31.42 gaze events and 7.42 gestures per video, while other datasets typically contain fewer than eight total events.Density varies with interaction structure, video duration, and visible social activity.
- B.5 Event Density and Training Distribution: Easy examples mainly test direct gaze or gesture recognition, while medium and hard examples require temporal or cross-modal reasoning; joint reasoning intentionally excludes easy questions.Joint questions relate gaze and gesture events or reason about person-level interaction dynamics.
- B.5 Event Density and Training Distribution: The dataset contains 145,049 gaze, 91,536 gesture, and 53,504 joint reasoning QA pairs, with 238,296 MCQ and 51,852 open-ended examples.MCQs support RL training, whereas open-ended examples support SFT.
- B.6 Comparison with Specialized Social VQA Benchmarks: GRASP differs from single-cue datasets by emphasizing temporal social-event dynamics such as joint attention and turn-taking, while reasoning-centric benchmarks provide cognitive insights at more limited scale.The comparison positions GRASP around fine-grained, participant-level non-verbal grounding.
- C.1 Training Details: Training proceeds by supervised fine-tuning on open-ended data followed by GRPO-based reinforcement learning on the multiple-choice split.Rewards combine answer correctness, output-format consistency, structural grounding, and social grounding.
- C.1 Training Details: SFT uses two epochs at 2 × 10−6, RL uses one epoch at 3 × 10−6, and GRPO uses K = 8 rollouts with a 0.05 KL penalty.Both stages sample videos at 2 fps and train in bfloat16 with AdamW, cosine scheduling, gradient clipping, and DeepSpeed ZeRO-2.
- C.2 Other Benchmark Details: Transfer evaluation uses MMSI, Online-MMSI, and TVQA+ to test person-reference tracking in multi-party conversation and spatio-temporally grounded video QA.MMSI includes Speaking Target Identification, Pronoun Coreference Resolution, and Mentioned Player Prediction tasks.
- C.2 Other Benchmark Details: TVQA+ provides 29,383 multiple-choice questions from 4,198 clips with 148,468 annotated images, 310,826 bounding boxes, and 2,527 visual concept categories.It combines subtitle-and-video question answering with relevant-moment localization and visual grounding of people and objects.
C.3 Participant-ID Corruption Analysis · C.4 Reasoning Length Breakdown
Participant-ID corruption substantially reduces SGR-trained models’ performance, especially for interactions requiring identity resolution, indicating reliance on video-grounded participant reasoning. Across social reasoning tasks, accuracy is uncorrelated with reasoning length, while the proposed method shortens reasoning and improves accuracy.
- C.3 Participant-ID Corruption Analysis: SGR’s participant-ID corruption test remaps Person IDs in questions, options, and metadata while preserving videos and correct answer letters.The test is designed to diagnose whether models exploit repeated textual participant IDs instead of grounding their reasoning in the video.
- C.3 Participant-ID Corruption Analysis: 50.4% to 32.6%: Qwen3-VL-8B+SGR drops substantially under participant-ID corruption.The corruption leaves the video unchanged while altering participant identities in the textual inputs.
- C.3 Participant-ID Corruption Analysis: 52.6% to 32.0%: Qwen3.5-9B+SGR likewise drops substantially under participant-ID corruption.The degradation is strongest in categories requiring specific participant resolution, including gaze following, temporal gaze reasoning, gesture recognition, reciprocal gestures, and joint person dynamics.
- C.3 Participant-ID Corruption Analysis: T4 drops by 22.4pp and 29.9pp, G4 drops by 27.2pp and 35.5pp, and J4 also shows a large degradation under corruption.These categories require tracking who looks at whom, who gestures to whom, or who participates in joint gaze-and-gesture events.
- C.3 Participant-ID Corruption Analysis: Categories with weaker dependence on participant identity, including mutual gaze duration and some group-attention questions, show smaller or mixed changes.By contrast, identity-dependent gaze, gesture, and joint-event categories consistently show large drops, though the diagnostic does not rule out every form of reward gaming.
- C.4 Reasoning Length Breakdown: Pearson r = + 0.039: accuracy and average reasoning length show no correlation across baseline models on GRASP-Bench social reasoning tasks.This contrasts with findings on other VQA tasks where longer reasoning helps performance.
- C.4 Reasoning Length Breakdown: The proposed method simultaneously shortens reasoning length and improves accuracy, showing that longer reasoning is not necessarily better reasoning here.This conclusion follows the observed lack of correlation between accuracy and average reasoning length.
C.5 SGR Error Profile … D.3 Failure Cases
The section analyzes SGR’s error profile across task difficulty and supplements quantitative results with qualitative comparisons, benchmark examples, reasoning traces, and representative failures. These analyses show that SGR follows benchmark difficulty while recovering some hard cases, yet remains vulnerable to ambiguous gestures and multiple gaze events.
- C.5 SGR Error Profile: SGR accuracy tracks difficulty buckets defined by how many base models answer correctly, revealing benchmark complexity beyond what the reward captures.The buckets estimate GRASP-Bench question difficulty for off-the-shelf comparison models and support analysis of SGR’s errors.
- C.5 SGR Error Profile: SGR recovers some cases that all other models miss, but fails on some easier cases that all other models answer correctly.The paper contrasts hard recoveries, such as the example in Fig. 1, with easier failures discussed in Appendix D.3.
- C.5 SGR Error Profile: The two SGR models exhibit similar overall error profiles while making different errors, motivating future analysis of model-specific error characteristics.Their differing errors create an opening for investigating how SGR-supported models fail.
- D Qualitative Comparison: Qualitative examples show how GRASP-Bench grounds questions in fine-grained gaze and gesture events, compares reasoning over shared interactions, and exposes remaining failures.These examples complement the paper’s main quantitative results.
- D.1 GRASP-Bench Examples: Figures 23–25 cover gaze reasoning T1–T6, gesture reasoning G1–G6, and joint gaze–gesture reasoning J1–J4 using tracked participant IDs and question–answer pairs.The examples span direct perception questions such as gaze-target and gesture-type recognition and harder reasoning questions.
- D.2 Reasoning Trace Comparison with Baselines: Reasoning traces compare SGR-trained models with supervised, reasoning, and RL post-training baselines across easy, medium, and hard GRASP-Bench examples.Additional examples cover MMSI’s STI, PCR, and MPP tasks and one TVQA+ example.
- D.3 Failure Cases: Failure cases include ambiguous reaching-versus-pointing gestures and intervals containing multiple gaze events from the same participant with several people involved.The gesture ambiguity persists even under human inspection because later frames clarify the reaching/recoil pattern.
E Broader Impacts
GRASP and SGR aim to advance multimodal social understanding by grounding reasoning in participant-level gaze and deictic-gesture interactions, with potential applications in human-centered and socially aware systems. The benchmark emphasizes event-level reasoning rather than identity or private-state inference, while acknowledging source bias, annotation noise, and model-induced limitations.
- Intended benefits: GRASP and SGR encourage models to reason about who interacts with whom from gaze and deictic gestures, supporting human-centered video understanding and socially aware systems.The stated application areas include assistive agents, collaborative robotics, and socially aware interfaces requiring participant-level grounding.
- Scope and safeguards: GRASP uses anonymous participant IDs and focuses on event-level questions such as who looks at whom or gestures toward whom, not individual identification, profiling, or private mental-state inference.Its sources are publicly available or research-oriented videos, and gaze and gesture interpretation can be ambiguous.
- Limitations: The dataset and models may inherit biases from source videos and foundation models because represented settings, camera styles, gestures, and interaction norms may not cover all populations or contexts.The paper identifies limited diversity as an ongoing concern rather than claiming broad representativeness.
- Limitations: Automatic gaze and gesture extraction can introduce annotation noise in complex scenes, mitigated through confidence filtering, structured validation, and subset human feedback.Improving diversity and event reliability remains future work.
- Transparency: Ground-truth answers are derived from participant IDs, event types, timestamps, and gaze–gesture links rather than free-form generated text to reduce unsupported inferences and enable more transparent study.The authors present GRASP as a documented research benchmark for socially grounded MLLMs.