GRASP: Learning to Ground Social Reasoning in Multi-Person Non-Verbal Interactions

• Gaze and gesture can signal who is addressed, what is referenced, and how participants relate, but interpreting these cues requires reasoning from ambiguous visual evidence.Source: §1 · p. 1
• Prior methods detect gaze or gesture as individual cues, while MLLMs often fail to organize them into temporally grounded participant interactions and instead rely on scene-level or linguistic shortcuts.Effective understanding requires grounding reasoning in gaze, deictic gestures, and their coordination across participants.Source: §1 · pp. 1–2, Figure 1 · p. 2
• GRASP contains 290K QA instances across 16 categories, grounded in 245K interaction events from 46K videos totaling 749 hours, alongside a 1K-item GRASP-Bench.A bottom-up pipeline detects gaze and deictic gestures, organizes them into social events, and models their joint compositions.Source: §1 · pp. 2–3
• Social Grounding Reward checks whether a model’s reasoning references the correct participants, directly encouraging evidence-aware reasoning about social events.It is proposed as a simple learning signal rather than a complex reward function.Source: §1 · pp. 2–3
• Post-training with GRASP data and policy achieves competitive GRASP-Bench performance, generalizes to three social reasoning benchmarks, and includes analyses of where social grounding helps.The introduction reports transfer across task formulations but gives no benchmark scores here.Source: §1 · pp. 2–3

Gaze and Gesture in Human Social Behavior

• Gaze research spans target estimation and social dynamics, but specialized gaze models remain dominant; this work equips MLLMs to use gaze for higher-level social understanding [13] [17].Methods have progressed from fusion-based systems using depth, pose, and head crops to decoding with visual foundation model representations [26] [72] [69].Source: §2 · p. 3
• Gesture datasets and foundation-model approaches cover diverse settings, but jointly modeling gaze and gesture to infer social intention remains a critical gap [7] [48].Source: §2 · p. 3

Multi-modal Large Language Models in Social AI

• Despite MLLM video QA advances [21] [22] [29], language-model reliance can produce visually ungrounded social answers, and broad benchmarks often miss fine-grained interpersonal cues [44] [41] [40] [56] [37] [45] [46] [74] [86] [76].The paper introduces a benchmark and post-training strategy to enhance MLLMs’ fine-grained social intelligence.Source: §2 · p. 3

3.1 Data Collection Pipeline

• Across 46K videos, GRASP unifies person-anchored gaze and deictic gesture annotations into a social graph containing 245K events.Each event records its type, participants, and temporal boundaries; gaze and gesture events can be linked when they co-occur, enabling joint-modality reasoning.Source: §3 · p. 4, §3.1 · p. 5
• The pipeline yields 169K gaze events across five types and 76K deictic gesture instances with initiator and target identities.Gaze types include mutual gaze, joint attention, gaze following, attention capture, and sudden gaze shift; gesture types include pointing, showing, giving, and reaching.Source: §3.1 · p. 5

3.2 Social Reasoning QA Taxonomy

• The 16-category taxonomy spans gaze reasoning (T1–T6), gesture reasoning (G1–G6), and joint gaze–gesture reasoning (J1–J4).Tasks range from cue perception and classification to temporal, reciprocal, sequence, group-attention, and cross-modal interaction reasoning.Source: §3.2 · p. 5
• Structured event metadata generates 290K QA pairs, with human checks of a subset for event consistency, answer verifiability, and ambiguity.Questions target participant identities, time intervals, and interaction types, and are anchored to an event with surrounding-frame context.Source: §3.2 · pp. 5–6

4 Learning to Ground Social Reasoning

• GRPO optimizes groups of reasoning trajectories using their rewards, with outputs separating intermediate <think> reasoning from final <answer> responses.The objective also uses a fixed reference policy and KL regularization.Source: §4 · p. 6

4.1 Training Strategy for Social Reasoning

• Training first SFT-warms on 52K GRASP examples, then applies reinforcement learning with correctness, format, structural-grounding, and social-grounding rewards.SFT targets enumerate temporally grounded gaze and gesture events in separate tags before giving the answer; this addresses sparse rollouts that otherwise fail to reference relevant cues.Source: §4.1 · p. 6

4.2 Social Grounding Reward for Social Learning Signal

• The social grounding reward compares predicted participant identities in gaze and gesture traces with target-event participants using weighted precision and recall.Balancing coverage and selectivity penalizes both mentioning irrelevant visible people and missing key participants; a structural term also encourages explicit dual-tag articulation of non-verbal observations.Source: §4.2 · pp. 6–7

5 Experiments

• Evaluation spans 16 GRASP-Bench categories and three public benchmarks, comparing proprietary, instruction-tuned, and RL-based MLLMs.MMSI and Online-MMSI include speaker-target identification, pronoun-coreference resolution, and mentioned-player prediction; Online-MMSI uses only past frames.Source: §5 · p. 7

5.1 Main Results on GRASP-Bench

• Socially grounded RL consistently outperforms supervised fine-tuning and existing RL post-training across GRASP-Bench reasoning types, especially gaze, gesture, and cross-modal reasoning.The results support grounding reasoning in nonverbal evidence rather than relying on coarse visual perception or language priors.Source: §5.1 · pp. 7–8

5.2 Generalization to Other Social Understanding Datasets

• In zero-shot evaluation on MMSI, Online-MMSI, and TVQA+, SGR-trained models outperform or remain competitive with instruction-tuned backbones and RL post-training baselines.Training uses GRASP only; the benchmarks cover social-reasoning tasks and spatiotemporally grounded video QA. The supplied table caption identifies these evaluation scopes but provides no scores.Source: §5 · p. 7, §5.2 · p. 8, Table 2 · p. 8

5.3 Ablation and Diagnostic Analysis

• GRPO adds moderate gains over the baseline, while adding SGR produces substantially larger gains across all metrics; per-category results show more consistent improvement than GRPO.The analysis attributes the difference to SGR explicitly grounding reasoning in nonverbal cues, whereas GRPO alone does not ensure reasoning aligns with relevant social evidence.Source: §5.3 · p. 9, Figure 4 · p. 9
• With SGR, average accuracy rises from 46.2 to 50.4 and from 45.3 to 52.6 across the two baseline backbones as model size increases.Scaled SGR models remain in the high grounded-participant-precision region while accuracy strengthens.Source: Table 4 · p. 8, §5.3 · p. 9, Figure 5 · p. 9
• Grounding precision correlates positively with accuracy (ρ = 0.64), and SGR achieves higher precision with fewer participant mentions than baselines that often mention irrelevant people.This selective participant grounding contrasts with RL baselines that can attain moderate accuracy despite low precision and noisy traces.Source: §5.3 · p. 9, Figure 5 · p. 9

6 Discussion and Conclusion

• GRASP provides 290K QA pairs across 46K videos and a 16-category taxonomy; SGR improves GRASP-Bench performance while preserving zero-shot performance on related social video QA tasks.Annotations can remain noisy in ambiguous scenes despite quality control and human validation. SGR mainly verifies participant-level grounding; stricter temporal alignment and richer social cues remain future work.Source: §6 · pp. 9–10

A.1 Video Pre-processing and Person-Consistent Gaze Trajectories

• An identity-aware pipeline tracks people, associates detected faces with person identities, and aligns each participant with a person box, face box, gaze point, and validity flag.Person tracks use SAM 3, and face-to-person matches are made with Hungarian matching; this identity-preserving representation supports later gaze features and event detection.Source: §A.1 · p. 15

A.2 Gaze Feature Extraction

• Gaze trajectories use conservative gap filling, while gaze changes are measured relative to each person’s head position for robustness to camera pans and global motion; median-distance scoring resists outliers.Gaps up to 3 frames are linearly interpolated; 4–10-frame gaps carry forward the last valid observation with decayed confidence, while longer gaps remain missing. Interpolation is blocked when the gap exceeds 3 seconds or face displacement exceeds 30% of frame width.Source: §A.2 · pp. 15–16

A.3 Social Gaze Event Detection

• Five gaze event classes—sudden gaze shift, joint attention, gaze following, attention capture, and mutual gaze—are detected from identity-consistent trajectories using event-specific temporal and participant criteria.Gaze following requires a 1.0–2.0-second lag, mutual gaze requires bidirectional face-box hits using measured gaze, and retained gaze events have confidence at least 0.9.Source: §A.3 · p. 16

A.4 Deictic Gesture Annotation

• Gesture predictions undergo verification scored out of 7 points, covering timeline alignment and visual-context accuracy; scores at most 5/7 trigger regeneration using the prior prediction and verification report as feedback.Retained gestures have confidence at least 0.85 and are stored in structured fields, including gesture type, initiator identity, target identity when applicable, and temporal boundaries.Source: §A.4 · p. 17

A.5 Unified Social Graph Construction

• Gaze and gesture annotations are merged into temporally ordered social-event nodes that record event type, participants, timing, confidence, and modality-specific attributes; linked gaze–gesture pairs support joint reasoning.Pairs within 3.0 seconds are protected during pruning, and graphs with many events are capped at 25 while prioritizing protected pairs, event diversity, and confidence.Source: §A.5 · p. 17

A.6 Social Reasoning QA Generation

• QA generation adapts question difficulty to event density across gaze, gesture, and joint categories, and generates joint questions only when temporally linked gaze–gesture pairs exist.Answers remain verifiable from structured graph fields, with generated QA filtered for invalid formats, time ranges, person references, speculative language, and category–modality mismatches.Source: §A.6 · p. 18

A.7 Quality Control and Human Validation

• Confidence filtering, conservative gesture annotation, and graph construction checks reduce automatic annotation noise while preserving gaze–gesture pairs needed for joint reasoning.The protocol skips ambiguous gestures, avoids inferring intent from non-gesture cues, and removes invalid temporal boundaries, duplicate events, and malformed records.Source: §A.7 · p. 18
• 74.2% overall validity was reported for audited GRASP-Bench QA, with gesture at 78.9% and joint reasoning at 80.0%; gaze was comparatively more challenging.The paper attributes the greater difficulty to verifying fine-grained gaze direction, temporal ordering, and group attention in dense multi-person scenes.Source: Table 11 · p. 19, §A.7 · p. 19
• A graph-level health check verifies that each corpus event includes required fields for event type, participants, start and end times, and confidence.Source: §A.7 · p. 19
• Four evaluators randomly audited approximately 25% of GRASP-Bench in a category-balanced sample, judging each QA instance for visual support, temporal grounding, and lack of ambiguity.Evaluators reviewed the video clip, participant IDs, temporal context, category label, question, and answer options.Source: §A.7 · p. 19, Figure 6 · p. 19

B Detailed GRASP Data Statistics

• GRASP draws on multi-party TV dialogue [43], social-intelligence reasoning, multi-speaker conversation [60], Embody3D [57], and social-deduction gameplay [7], augmented with online videos for dense interaction cues.The online video sources augment the social-deduction domain.Source: B Detailed GRASP Data Statistics · pp. 19–20
• Across 46,158 processed videos, identity-anchored gaze trajectories and deictic gestures yield 290,148 QA pairs in 16 categories.The final training data contains 238,296 multiple-choice QA pairs for RL training and 51,852 open-ended QA pairs for SFT training.Source: B Detailed GRASP Data Statistics · p. 20

B.1 Per-Dataset Scale

• Social Deduction Game contributes 127,033 gaze events and 30,006 gestures from 4,044 videos, reflecting dense multi-person interactions.Source: §B.1 · p. 20, Figure 7 · p. 20

B.3 QA Category Distribution

• The QA taxonomy divides 16 categories into gaze, gesture, and joint reasoning, including cross-modal temporal questions.Gaze and gesture each have six categories, while joint reasoning has four. Joint questions relate gaze and gesture over time, including temporal alignment, gaze response to gesture, eye contact during interaction, and cross-modal person dynamics.Source: Figure 9 · p. 21, §B.3 · p. 22

B.2 Gaze and Gesture Event Distributions

• Gaze following dominates retained gaze events (31.2%), while pointing dominates detected gestures (52.5%).Retained gaze events cover gaze following, attention capture, joint attention, mutual gaze, and sudden gaze shift; gesture annotations cover pointing, reaching, showing, and giving. Events are filtered at confidence ≥0.9 for gaze and ≥0.85 for gestures.Source: Figure 8 · p. 20, §B.2 · p. 21

B.4 Construction Yield

• Filtering retains 215,378 of 429,663 raw gaze detections (50.1%), then gaze and gesture events merge into 181,926 timeline events before QA generation.A timeline event can support multiple question forms, including single-event recognition, temporal comparison, sequence reasoning, and cross-modal reasoning, so QA-pair count exceeds timeline-event count.Source: Figure 10 · p. 21, §B.4 · p. 22

B.5 Event Density and Training Distribution

• Social Deduction Game has the highest density at 31.42 gaze events and 7.42 gestures per video, while other source datasets typically have fewer than eight total events per video.The passage attributes variation to differences in interaction structure, video duration, and visible social-cue frequency.Source: §B.5 · p. 22
• Easy MCQs mainly test direct gaze or gesture recognition, whereas medium and hard examples require temporal or cross-modal reasoning; joint reasoning has no easy questions by design.Open-ended examples are excluded from the training-distribution heatmap because they are used for SFT and categorized as hard reasoning examples.Source: §B.5 · p. 22, Figure 12 · p. 23
• GRASP contains 145,049 gaze, 91,536 gesture, and 53,504 joint-reasoning QA pairs; its 238,296 MCQ examples support RL, while 51,852 open-ended examples support SFT.Source: §B.5 · p. 22

B.6 Comparison with Specialized Social VQA Benchmarks

• GRASP pairs temporal social-event reasoning with 46K videos and gaze/gesture grounding, unlike single-cue GazeVQA and SocialGesture [7] or scale-limited R3-VQA and Social Genome [56].The comparison frames GRASP as supporting MLLM alignment with nuanced social behaviors.Source: §B.6 · p. 23, Table 12 · p. 23

C.1 Training Details

• Training first uses SFT on open-ended GRASP data to establish structured reasoning, then GRPO-based RL on the multiple-choice split with correctness, format, structural-grounding, and social-grounding rewards.Source: §C.1 · pp. 23–24
• Both stages sample videos at 2 fps and use bfloat16, AdamW, cosine scheduling, gradient clipping, and ZeRO-2; SFT runs two epochs at 2 × 10−6 and RL one at 3 × 10−6.GRPO uses K = 8 rollouts, a 0.05 KL penalty, mean-centered advantage normalization, ±5.0 clipping, and reward weights 1.0/0.1/0.05/0.2 for accuracy/format/structural/social grounding.Source: §C.1 · p. 24

C.2 Other Benchmark Details

• MMSI and Online-MMSI evaluate person-reference tracking in multi-party conversation; Online-MMSI limits input to context before the target utterance, while TVQA+ tests grounded human-centered video QA.MMSI includes 151 YouTube games and 48 Ego4D games, with STI, PCR, and MPP tasks; TVQA+ has 29,383 multiple-choice questions from 4,198 clips with spatial grounding annotations.Source: §C.2 · p. 24
• The participant-ID corruption test remaps IDs in questions, answer options, and metadata while leaving video and correct answer letters unchanged; Normal and Corrupt are accuracy, and Drop is their difference in percentage points.Source: Table 14 · p. 25

C.3 Participant-ID Corruption Analysis

• The test remaps textual Person IDs while leaving the video unchanged, disrupting alignment but preserving answer-position priors and textual format.If models relied mainly on text-only ID shortcuts, their accuracy should remain stable; visually grounded models should degrade when mapping participant references to people in the video is required.Source: §C.3 · p. 25
• Under participant-ID corruption, both SGR-trained models drop substantially: Qwen3-VL-8B+SGR falls from 50.4% to 32.6%, while Qwen3.5-9B+SGR falls from 52.6% to 32.0%.Source: §C.3 · p. 25
• Corruption produces large drops in identity-dependent categories: T4 by 22.4pp and 29.9pp, G4 by 27.2pp and 35.5pp, and J4 by 29.3pp and 11.4pp, respectively.The two values in each pair correspond to Qwen3-VL-8B+SGR and Qwen3.5-9B+SGR, respectively.Source: §C.3 · p. 25
• Mutual-gaze-duration and some group-attention questions show smaller or mixed changes, unlike tasks tracking who looks or gestures at whom or joins both events.The diagnostic addresses the ID-repetition shortcut but does not rule out every form of reward gaming; the final SGR policies depend on consistency between textual references and visually grounded identities.Source: §C.3 · p. 25

C.4 Reasoning Length Breakdown

• Across baseline models on GRASP-Bench, accuracy and average reasoning length show no correlation (Pearson r = + 0.039).Figure 13 plots accuracy against average reasoning length for all baselines.Source: §C.4 · pp. 25–26, Figure 13 · p. 26
• SGR simultaneously shortens reasoning length and improves accuracy, showing that longer reasoning is not necessarily better reasoning on these social reasoning tasks.This contrasts with findings for other VQA tasks where longer reasoning helps performance.Source: §C.4 · p. 26

C.5 SGR Error Profile

• SGR performance tracks base-model difficulty, suggesting benchmark complexity includes factors the reward does not capture.Cases are grouped by how many base models answer correctly, and model accuracy is computed for each bucket.Source: §C.5 · p. 26, Figure 14 · p. 26
• SGR recovers some cases all other models miss but fails on some they answer correctly, and the two SGR models make different errors.Their similar error profiles leave room to investigate how their errors differ.Source: §C.5 · pp. 26–27

D Qualitative Comparison

• Qualitative examples illustrate fine-grained cue grounding, model reasoning over shared interactions, and remaining failures.Source: D Qualitative Comparison · p. 27

D.1 GRASP-Bench Examples

• GRASP-Bench examples span gaze, gesture, and joint reasoning, from cue recognition to temporal ordering, reciprocal interaction, and gaze–gesture alignment.Each example presents tracked participant IDs, a non-verbal cue, and its question–answer pair.Source: §D.1 · p. 27

D.2 Reasoning Trace Comparison with Baselines

• Across GRASP-Bench, MMSI, and TVQA+ examples, baselines often rely on salient participants or dialogue priors, while SGR-trained models more consistently ground answers in relevant participants and visual/social cues.The examples compare reasoning traces across easy, medium, and hard cases, as well as three MMSI tasks and a TVQA+ example.Source: §D.2 · p. 27

D.3 Failure Cases

• Representative failures arise from ambiguous pointing-versus-reaching gestures and missed gaze cues amid dense multi-person activity.In one case, later frames clarify a reaching/recoil pattern; in another, multiple gaze events and nearby social activity lead to incomplete evidence.Source: §D.3 · p. 27

Broader impacts

• GRASP and SGR may benefit human-centered video understanding, assistive agents, collaborative robotics, and socially aware interfaces that need participant-level grounding.The work encourages models to reason about who interacts with whom from gaze and deictic gestures.Source: E Broader Impacts · p. 27

Scope and limitations

• GRASP uses anonymous participant IDs and event-level visual reasoning, not individual identification, profiling, or inference of private mental states, and should not support high-stakes decisions about people.Gaze and gesture can be ambiguous, culturally variable, or affected by occlusion and camera viewpoint.Source: E Broader Impacts · p. 27

Biases and reliability

• Source-video and foundation-model biases may limit representativeness, while automatic gaze and gesture extraction can add noise; confidence filtering, structured validation, and subset human feedback mitigate these issues.Improving data diversity and event reliability remains future work.Source: E Broader Impacts · pp. 27–28

Dataset construction and transparency

• Ground-truth answers derive from structured participant, event, timestamp, and gaze–gesture fields rather than free-form generated text, and the documented benchmark is intended to support transparent study of socially grounded MLLMs.GRASP uses multimodal and language models for gesture annotation, event description, and QA generation.Source: E Broader Impacts · p. 28