Source-linked AI summary
Human-Inspired Social Engagement Analysis via Interpretable Mutual Visual Attention
Urwa Fatima, Mohammad Zohaib, Francesca Odone, Nicoletta Noceti
TL;DR
Computational analysis of social interaction from non-verbal visual data remains limited beyond isolated recognition tasks. The paper introduces an interpretable modular framework that models dyadic visual attention and aggregates it into individual- and group-engagement measures, with quantitative evaluations and practical visualizations for non-technical users. Experiments demonstrate interaction-event detection and classification, while engagement indices and visualizations make group dynamics transparent to teachers, caregivers, and social workers.
Problem
Computational approaches using only non-verbal cues remain relatively limited and often focus on isolated tasks such as head pose or action recognition.
Method
The framework estimates head direction, infers dyadic visual attention, classifies interaction states, and aggregates them into interpretable individual- and group-engagement indices.
Results
The experiments demonstrate the framework’s ability to detect and classify interaction events and produce engagement indices and visualizations that make its decision process transparent.
Takeaways & Limitations
The framework provides teachers, caregivers, and social workers with visual tools for understanding interaction dynamics instead of only receiving engagement predictions.
Takeaways & Limitations
The design assumes that attention is an observable cue for engagement, reciprocal attention is stronger evidence than one-sided attention, and group engagement emerges from local dyadic interactions.
Abstract
from arXiv · showhide
Understanding social interactions from non-verbal visual data is important for behavior analysis and activity monitoring. We propose an interpretable computational model of social engagement inspired by psychological theories of mutual visual attention. Rather than learning interaction patterns end-to-end, our framework explicitly models dyadic visual attention and aggregates these cues into interpretable measures of individual and group engagement. The resulting modular framework combines state-of-the-art head orientation estimation with lightweight geometric reasoning, producing explanations that remain accessible to non-technical users. We evaluate the proposed approach on a variety of data through quantitative experiments and demonstrate its practical usefulness with qualitative visualizations designed to support teachers, caregivers, and social workers in understanding group interaction dynamics.
1 Introduction
The paper proposes an interpretable, modular framework that infers group engagement from dyadic visual attention and aggregates these relationships into individual- and group-level measures. It evaluates the framework quantitatively and qualitatively, including visualizations and a kindergarten demonstration for non-technical users.
- The approach addresses limited computational use of non-verbal cues beyond isolated tasks such as head pose estimation, body pose estimation, and action recognition.
- The framework models dyadic interactions from non-verbal visual cues before aggregating them into interpretable individual- and group-engagement measures.
- The pipeline estimates head direction, infers pairwise visual attention, classifies dyadic states, and computes engagement indices.Its modular stages keep intermediate social cues explicit rather than hidden inside an end-to-end model.
- The framework is evaluated quantitatively on interaction-pattern recognition and qualitatively across interaction scenarios, including a kindergarten cooperative activity.The authors present visualizations intended to help psychologists and educators study small-group interactions.
- The work contributes two engagement indices that quantify interaction at individual and group levels.
- An intuitive visualization interface makes multi-level social interaction analysis accessible to non-technical users.
2 Related Works
Prior work supports hierarchical reasoning from pairwise interpersonal dynamics to group-level properties, including engagement and interaction structure. The paper builds on attention-based cues while distinguishing its interpretable approach from newer joint deep-learning models that require large datasets and lack interpretability.
- The Social Relations Model treats the dyad as the fundamental unit of social behavior and decomposes group interactions into actor, partner, and relationship effects.
- Hierarchical computational frameworks integrate low-level behavioral cues to estimate pairwise relations and infer group properties such as engagement, cohesion, rapport, and leadership.
- Mutual facing and shared attention have been used to detect dyadic engagement and model group structure.
- Recent deep-learning approaches jointly reason over individuals and groups but require large datasets and lack interpretability.
- The proposed framework leverages head direction as a proxy for gaze direction and attention.
3 Proposed Framework
The framework mirrors hierarchical social reasoning by inferring engagement from dyadic visual attention and aggregating pairwise states into interpretable individual and group measures. It uses head-pose estimation, geometric reasoning, interaction classification, and weighted engagement indices in a modular pipeline.
- Design principles: The framework treats attention as an observable cue, reciprocal attention as stronger interaction evidence, and group engagement as emerging from local dyadic interactions.These principles motivate hierarchical reasoning rather than direct end-to-end engagement prediction.
- Stages 1–2: The pipeline extracts tracked facial keypoints, normalizes them for scale and face position, and uses HHP-Net to predict head orientation with uncertainties.Normalized keypoints are converted into 2D unit head-direction vectors anchored at the facial centroid.
- Stage 3: Pairwise geometric reasoning compares each person’s head direction with the direction toward another person to classify bidirectional, unidirectional, or non-interacting states.The angles ϕ12 and ϕ21 are thresholded by τ; both below τ indicates bidirectional attention, one indicates unidirectional attention, and neither indicates non-interaction.
- Stage 3: Pairwise predictions are reconciled by prioritizing bidirectional pairs, then assigning remaining members to unidirectional states, and finally classifying unassigned members as non-interacting.For unidirectional interactions, the person below τ is the looker and the other is the target.
- Stage 4: IEI weights bidirectional, unidirectional-looker, and unidirectional-target frames at 1.00, 0.30, and 0.05, respectively, producing a 0–100 individual engagement score.The index is computed over detected frames and gives mutual events the highest engagement weight.
- Stage 4: GES compares weighted predicted bidirectional and unidirectional engagement with maximum possible engagement across video frames, with βB = 1 and βU = 0.5.Its scale runs from 0 for no interaction to 100 for maximum possible interaction.
4 Experiments and Results
The evaluation combines quantitative interaction-classification benchmarks with qualitative analyses of engagement indices and visualizations across dyadic, triadic, and varied group-activity videos.
- Benchmarks: Three manually annotated CMU Panoptic Haggling sequences extend the comparison from dyads to triadic conversational scenes.Each sequence contains three face-to-face participants and lasts approximately 4–6 minutes.
- Benchmarks: Fifteen unannotated Pexels videos support qualitative evaluation across high-resolution group activities, ages, and indoor/outdoor settings.The videos last approximately 0.15–2 minutes.
- Visualization analysis: The visualization tool combines sequential frames and estimated head directions with temporal interaction states, partner IDs, frame-level counts, and individual IEI values.These views are reported for sequences from both Pexels and CMU.
- Engagement scenarios: GES varies across Pexels scenarios, from 31.8 when only a teacher interacts as a unidirectional looker to 80.6 during a cooperative family activity.A middle scenario has GES 68.7, while the reported 31.8 scene shows limited interaction beyond the teacher.
5 Conclusion
The paper presents a human-inspired, interpretable framework that derives individual and group engagement from dyadic visual attention cues. Its main scope limitation is reliance on 2D pose estimation, motivating future 3D and multimodal extensions.
- Contribution: The framework uses explicit intermediate representations to infer group engagement from dyadic visual attention rather than relying on a black-box predictor.It combines pose and head-direction estimation with explicit dyadic interaction modeling.
- Contribution: The framework derives interpretable engagement measures at both individual and group levels, supported by engagement indices and visualizations.The stated users include teachers, caregivers, and social workers studying interaction dynamics.
- Findings: Experiments demonstrate the ability to detect and classify interaction events while making the decision process transparent through engagement indices and visualizations.The visual outputs are intended to expose interaction dynamics instead of providing only engagement scores.
- Limitation: Reliance on 2D pose estimation may introduce ambiguities because three-dimensional scenes are projected onto the image plane.Future work is proposed around 3D human representations, multimodal cues, speaker attribution, and patterns beyond dyadic relationships.