Source-linked AI summary
Deep Reasoning with Knowledge Graph for Social Relationship Understanding
Zhouxia Wang, Tianshui Chen, Jimmy Ren, Weihao Yu, Hui Cheng, Liang Lin
TL;DR
Social relationship recognition from images requires contextual cues that previous studies largely overlook. The paper integrates correlations between relationships and semantic objects into an end-to-end Graph Reasoning Model using graph propagation and attention, and reports superiority over existing methods on two benchmarks.
Problem
Previous studies largely overlook contextual-object cues, their correlations with social relationships, and their interaction with the persons being recognized.
Method
The Graph Reasoning Model represents relationship-object correlations as a knowledge graph, propagates messages with GGNN, and attends to discriminative contextual objects.
Results
The GRM outperforms existing methods on the PISC and PIPA-Relation benchmarks, achieving 82.8% mAP for coarse-level recognition and 62.3% accuracy on PIPA-Relation.
Takeaways & Limitations
Explicitly reasoning over contextual objects provides an interpretable approach to social relationship recognition and improves performance when prior knowledge is incorporated.
Takeaways & Limitations
The GRM relies on Faster R-CNN semantic-object detections and incurs extra detection annotations during training and testing.
Abstract
from arXiv · showhide
Social relationships (e.g., friends, couple etc.) form the basis of the social network in our daily life. Automatically interpreting such relationships bears a great potential for the intelligent systems to understand human behavior in depth and to better interact with people at a social level. Human beings interpret the social relationships within a group not only based on the people alone, and the interplay between such social relationships and the contextual information around the people also plays a significant role. However, these additional cues are largely overlooked by the previous studies. We found that the interplay between these two factors can be effectively modeled by a novel structured knowledge graph with proper message propagation and attention. And this structured knowledge can be efficiently integrated into the deep neural network architecture to promote social relationship understanding by an end-to-end trainable Graph Reasoning Model (GRM), in which a propagation mechanism is learned to propagate node message through the graph to explore the interaction between persons of interest and the contextual objects. Meanwhile, a graph attentional mechanism is introduced to explicitly reason about the discriminative objects to promote recognition. Extensive experiments on the public benchmarks demonstrate the superiority of our method over the existing leading competitors.
1 Introduction
Social relationship recognition in still images requires interpreting person pairs together with contextual objects, a cue previous work largely overlooks. The paper proposes an end-to-end Graph Reasoning Model that incorporates structured knowledge and attention for this task.
- Social relationships are important for intelligent systems to understand human behavior and interact appropriately.
- Recognizing relationships from images is difficult because the same people may have different relationships across occasions with different contextual cues.Business wear and office supplies suggest colleagues, whereas household objects such as beds suggest family members.
- Previous approaches largely overlook contextual-object semantics, prior relationship correlations, and interactions between contextual objects and persons.
- The Graph Reasoning Model organizes correlations between social relationships and semantic objects as a structured graph, then uses neural message propagation to learn node-level features.
- The model introduces graph attention to reason explicitly about key contextual cues and provides an interpretable approach to recognition.
2 Related Work
Earlier research covers social relationship recognition and graph neural networks, but the paper combines knowledge-graph reasoning with learned message propagation for social relationship recognition. Its GRM uses contextual-object information and person-object interaction as a unified approach.
- Related vision work has used social information as supplementary cues for tracking, trajectory prediction, and group activity analysis.
- Other studies directly predict social roles and relationships, including kinship recognition based on facial appearance, attributes, and landmarks.
- Knowledge-graph methods represent correlations or object associations as graph structures for visual reasoning, often relying on handcrafted features and manually defined rules.
- Learnable graph networks such as Graph LSTM, Graph CNN, and GGNN propagate messages to model dependencies and learn node- or graph-level representations.
- Unlike prior graph-based vision methods, the GRM applies GGNN to prior-knowledge graphs to model interactions between person pairs and contextual objects for social relationship recognition.
3 Graph Reasoning Model
The Graph Reasoning Model represents social relationships and contextual objects in a structured graph, propagates messages between them, and uses graph-restricted attention to weight informative context for recognition.
- The graph contains social-relationship and object nodes, with edges representing co-occurrence probabilities.
- Relationship nodes use person-pair region features, while object nodes use features from detected image regions.Person-pair features combine union and individual regions with geometry information; detected object regions initialize corresponding category nodes.
- GGNN propagation repeatedly aggregates neighbor messages and updates each node through a gated mechanism.After T interactions, messages have propagated through the graph and final hidden states are obtained for all nodes.
- Graph attention addresses non-informative or interfering objects by adaptively selecting relevant contextual nodes.The method is motivated by objects such as desks that co-occur frequently with many social relationships.
- Attention coefficients are computed only for graph-neighbor object nodes, normalized with a sigmoid, and set to zero otherwise.The weighted object features are concatenated with each relationship node’s own feature before final recognition.
- The resulting feature vector is passed to a fully connected layer, and training uses cross-entropy loss.
4.1 Knowledge Graph Building
The knowledge graph requires paired annotations for social relationships and image objects, so the method uses a COCO-trained Faster R-CNN detector to obtain object annotations.
- Building the graph requires annotations of both social relationships for person pairs and objects present in images.
- Because the social relationship datasets lack these object annotations, a Faster R-CNN detector trained on COCO detects semantic objects.
4.2 Experiment Setting
Experiments evaluate GRM on PISC and PIPA-Relation, with PISC supporting coarse- and fine-level relationship recognition.
- Datasets: The evaluation uses the PISC and PIPA-Relation datasets.PISC contains 22,670 images.
- Tasks: PISC defines coarse-level recognition over three relationship categories.The categories are No Relation, Intimate Relation, and Non-Intimate Relation.
- Tasks: PISC also defines fine-level recognition over six relationship categories.The categories are Friend, Family, Couple, Professional, Commercial, and No relation.
- Protocol: The study follows the standard PISC train, validation, and test split.The passage specifies separate image and relationship-instance counts for coarse- and fine-level tasks.
4.3 Comparisons with State-of-the-Art Methods
GRM is compared with established CNN and contextual methods on PISC and PIPA-Relation. It achieves the strongest reported results on both benchmarks while using detected semantic objects to build and initialize graph context.
- Comparison protocol: GRM is compared with baseline and state-of-the-art methods on both PISC and PIPA-Relation.The PISC comparisons include Union CNN, Pair CNN variants, and Dual-glance.
- PISC results: 82.8% mAP on coarse-level PISC recognition improves the previous best method by 3.1%.The same evaluation reports per-class recall and mAP over relationships.
- PISC results: 68.7% mAP on fine-level PISC recognition improves the previous best method by 5.5%.The reported improvement is larger for fine-level than coarse-level recognition.
- Implementation scope: GRM uses Faster R-CNN semantic-object detections to build the knowledge graph and initialize contextual object nodes.The comparison notes that related methods also incur extra detection annotations.
- PIPA-Relation results: 62.3% accuracy on PIPA-Relation beats the previous best method by 2.7%.The comparison includes Two stream CNN and an implemented Dual-Glance baseline.
4.4 Ablation Study
Ablations test the knowledge graph, graph attention, and object-detection threshold. Performance drops when graph structure or learned attention is removed or randomized, while a relative threshold of 0.3 performs best.
- Significance of knowledge graph: Randomizing the graph adjacency scores reduces mAP from 82.8% to 81.4% on coarse-level recognition and from 68.7% to 63.5% on fine-level recognition.The experiment retrains GRM with a randomly initialized adjacency matrix on PISC.
- Graph attention mechanism: Removing graph attention causes an obvious mAP drop, especially for fine-level recognition.The no-attention baseline concatenates relationship-node and all object-node features.
- Graph attention mechanism: Random attention scores perform worse than using all node features because they may select non-informative or interferential objects.This baseline replaces learned attention coefficients and retrains the model identically.
- Detection threshold: A relative object-detection threshold of 0.3 produces the best results among tested threshold values.Lower thresholds may add false detections, whereas higher thresholds may miss contextual cues.
4.5 Qualitative Evaluation
Qualitative examples show GRM assigning attention to contextual objects that distinguish social relationships, rather than treating all detected objects as equally informative.
- Example 1: GRM highlights laptop and other office supplies as cues for recognizing a professional relationship.Desk and cup are described as frequent across relationships and therefore less discriminative.
- Example 2: GRM attends to bowl and pizza as key cues for recognizing a friend relationship.The examples visualize person regions, detected objects, ground-truth relationships, predictions, and top-scoring object nodes.
5 Conclusion
The proposed Graph Reasoning Model incorporates common-sense correlations between social relationships and contextual objects into a deep neural network. Its propagation and attention mechanisms explore person–object interactions and select discriminative objects, outperforming existing state-of-the-art methods on two benchmarks.
- GRM incorporates common-sense correlations between social relationships and semantic contextual cues into a deep neural network.
- A propagation model explores interactions between the person pair of interest and contextual objects by propagating node messages through the graph.
- A graph attention module measures node importance and adaptively selects discriminative objects to aid social relationship recognition.
- GRM outperforms existing state-of-the-art methods on the PISC and PIPA-Relation large-scale benchmarks.