Understanding Human Gaze Communication by Spatio-Temporal Graph Reasoning

• Gaze communication can convey hidden mental states and augment verbal interaction, motivating systematic study alongside natural language processing.Source: §1 · p. 1
• The paper distinguishes six atomic gaze classes and five higher-level events formed from temporal compositions of those atomic patterns.Joint Attention typically progresses through mutual, refer, and follow gaze, then cycles back to mutual gaze.Source: §1 · p. 2
• VACATION supports prediction of atomic- and event-level gaze communication from third-person social videos with face and object bounding boxes.The task requires modeling human-human and human-scene interactions across space and time.Source: §1 · p. 2
• A spatio-temporal reasoning graph network propagates information over a social graph for atomic-level detection, while an encoder-decoder event network models temporal coherence.The event network is intended to eliminate noise in gaze communications.Source: §1 · p. 2
• The paper’s contributions comprise the new task, the annotated VACATION dataset with benchmarks, and networks for hierarchical atomic- and event-level reasoning.Source: §1 · p. 2

2.1. Gaze Communication in HHI

• In human-human interaction, gaze conveys information and helps regulate intimacy, turn-taking, conversational pace, and social or emotional states; computer-vision work has rarely studied gaze communication.Prior vision work mainly models salient natural-environment properties, with only a few studies of shared attention in social scenes.Source: §2.1 · p. 3

2.2. Gaze Communication in HRI

• Human-robot interaction research incorporates responsive gaze for familiarity and trust, including mutual, follow, and refer gaze and joint attention in assistive settings.Source: §2.2 · p. 3

2.3. Graph Neural Networks

• Graph neural networks pass local messages to represent relations among nodes; this work applies that approach to a spatio-temporal social graph for gaze communication.The cited applications include structured classification, relational reasoning, multi-agent communication, and human-object interaction.Source: §2.3 · p. 3

3. The Proposed VACATION Dataset

• VACATION contains 300 social videos featuring diverse gaze communication behaviors.Source: §3 · p. 3

3.1. Data Collection

• The 300 high-quality sequences span 96,993 frames and 3,880 seconds, covering varied daily scenes, cultures, actors, costumes, and settings.Videos range from 2.2 to 74.56 seconds and average 13.28 seconds; source clips were gathered from more than 50 TV shows and movies.Source: §3.1 · p. 3

3.2. Data Annotation and Statistics

• Annotations include face and key-object boxes, each person’s attention target, and atomic- and event-level gaze labels, with cross-validation by two volunteers and specialist adjudication.The dataset includes 206,774 face boxes and 85,441 key-object boxes; annotation took about 1,616 hours.Source: §3.2 · p. 4
• The dataset is split into 180 training, 60 validation, and 60 test videos, with no source overlap across sets.The split contains 57,749 training frames, 22,005 validation frames, and 17,239 test frames.Source: Table 2 · p. 4

4. Our Approach

• The model uses a spatio-temporal graph network for atomic gaze inference and an encoder-decoder event network for event-level prediction.Graph nodes represent humans and the social scene; edges represent gaze communication structures.Source: §4 · p. 4

4.1. Model Formulation

• The social graph is complete and directed, with human and scene nodes connected by edges for possible human-human or human-scene relations.Human nodes receive one of six atomic gaze labels.Source: §4.1 · p. 5
• Unlike graph neural networks focused only on graph- or node-level labels, the model learns graph structure and all human-node gaze labels simultaneously.Source: §4.1 · p. 5
• Spatial message passing iteratively updates edge representations, infers graph connectivity, and updates node representations using information propagated over the learned graph.Connectivity weights the incoming edge information used in node updates.Source: §4.1 · p. 5
• An LSTM applies temporal reasoning to spatially updated human-node representations before a readout classifier predicts each node’s gaze label.The temporal module uses each node’s representation over time; the readout maps node features to the atomic label space.Source: §4.1 · p. 6
• The event network encodes atomic-gaze transition and frequency vectors separately, then decodes their concatenation to predict the event label.The encoder-decoder is intended to reduce noise in atomic gaze communications.Source: §4.1 · p. 6

4.2. Detailed Network Architecture

• For explicit graph learning, annotated human-human and human-scene relations supervise adjacency, while incoming connectivity to the dummy scene node is set to zero.Node features come from ResNet50 fc7 features for each human bounding box and the whole frame for the scene node.Source: §4.2 · p. 7

5.1. Experimental Setup

• Evaluation uses precision, F1-score, top-1 and top-2 average accuracy, with baseline models and component variants for comparison.The atomic temporal reasoning module uses a five-frame sequence; ablations vary node features, adjacency branches, message-passing iterations, and temporal reasoning.Source: §5.1 · pp. 7–8

5.2. Results and Analyses

• Figure 5 shows correct atomic-label predictions and errors; reported failure factors include ambiguous or illegible eyes and rapid, short gaze phases.Source: Figure 5 · p. 8, §5.2 · p. 8
• 55.02% top-1 average accuracy is the full model’s best atomic-level test result; the event network reaches 37.1% with predicted atomic labels and 55.9% with ground-truth labels.The event-level quantitative results are presented in Table 4.Source: §5.2 · p. 8, Table 4 · p. 8
• Message passing generally improves performance as iterations increase, but performance drops slightly beyond a certain extent.Source: §5.2 · p. 8

• The paper addresses inferring human gaze communication at atomic and event levels in third-person social videos, and presents VACATION and benchmark results.Source: §6 · p. 8
• The model combines graphs and standard feedforward neural networks to capture gaze patterns and provide better compositionality.Source: §6 · p. 8
• The authors present the work and dataset as resources intended to facilitate future studies of gaze communication.Source: §6 · p. 8