Source-linked AI summary
Learning Social Relation Traits from Face Images
Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang
TL;DR
The paper investigates whether fine-grained, psychologically grounded social relation traits can be characterized and quantified from face images. It proposes a deep model with rich attribute-driven face representations, pairwise reasoning, and a bridging layer for heterogeneous datasets with missing labels. Experiments show effective fine-grained social relation learning in images and videos.
Problem
The study addresses the challenge of predicting high-level social relation traits from face images when prediction depends on complex facial attributes and heterogeneous datasets lack complete labels.
Method
The method learns face representations from gender, expression, head pose, and age attributes, then performs pairwise-face reasoning using a bridging layer across heterogeneous datasets.
Results
70.6% accuracy was achieved with expression-attribute pre-training, compared with 64.0% without it; all deep-model variants outperformed HOG+SVM.
Takeaways & Limitations
The study demonstrates effective fine-grained social relation learning from face images and videos using psychologically grounded relation traits.
Abstract
from arXiv · showhide
Social relation defines the association, e.g, warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterised and quantified from face images in the wild. To address this challenging problem we propose a deep model that learns a rich face representation to capture gender, expression, head pose, and age-related attributes, and then performs pairwise-face reasoning for relation prediction. To learn from heterogeneous attribute sources, we formulate a new network architecture with a bridging layer to leverage the inherent correspondences among these datasets. It can also cope with missing target attribute labels. Extensive experiments show that our approach is effective for fine-grained social relation learning in images and videos.
1. Introduction
The study asks whether fine-grained social relation traits can be inferred from jointly viewed face images. It addresses rich facial attributes and heterogeneous, incompletely labeled datasets with a deep pairwise-reasoning model.
- Motivation: Social relation traits such as friendliness, warmth, and dominance are studied as high-level properties inferable from face images.The intended applications include relation mining in social networks, personal albums, and films.
- Challenges: Accurate relation prediction requires a rich face representation capturing attributes including expression, head pose, gender, and age.Psychological studies motivate linking facial factors to interpersonal relations.
- Challenges: No single dataset provides all required facial-attribute labels, while available datasets differ in annotations, environments, and statistical distributions.Some datasets provide expression labels, whereas others provide only gender labels.
- Approach: The proposed architecture handles missing attribute labels and bridges heterogeneous datasets using weak constraints from associated face-part appearances.The model learns face representations driven by expression, pose, gender, and age attributes.
- Contributions: The study contributes a psychologically grounded pairwise relation task, a new labeled social-relation dataset, and a multi-task architecture for heterogeneous sources.The architecture can also incorporate relative face location as an additional cue.
2. Related Work
Prior work addresses social signals, group behavior, and face analysis, but generally does not infer fine-grained interpersonal traits directly from face pairs. This study targets abstract relations requiring joint reasoning across multiple subjects and facial attributes.
- Social relation analysis: Existing social-relation studies often infer coarse connections, interactions, or social roles rather than fine-grained interpersonal traits.Examples include social-group discovery, three interaction classes, and roles such as father-child or husband-wife.
- Affective analysis: Affective-analysis methods usually examine one person and prototypical expressions, limiting their direct use for interpersonal relation inference.The target relations also involve attributes such as age and gender.
- Group behavior: Group-behavior studies mainly recognize action-oriented behaviors such as hugging, handshaking, or walking rather than abstract relations from faces.These studies often exploit spatial configuration and actions.
- Deep learning: Deep learning has succeeded in several face-analysis tasks but had not been adopted for face-driven social-relation mining requiring joint reasoning from multiple subjects.The proposed work addresses complex facial attributes using heterogeneous datasets and face-pair learning.
3. Social Relation Prediction from Face Images
The study predicts eight psychologically grounded, fine-grained social relation traits from face pairs using a deep network that combines rich facial attributes, spatial cues, and pairwise reasoning. It addresses heterogeneous attribute datasets, missing labels, and cross-dataset distribution differences through attribute-driven pretraining and a correspondence-based bridging layer.
- 3.1. Definitions of Social Relation Traits: The interpersonal circle defines 16 segments grouped into eight non-exclusive binary relation traits that can co-occur in an image.The study examines whether these traits are detectable and quantifiable from face images.
- 3.2. Social Relation Dataset: 8,306 web and movie images form a pairwise social-relation dataset with face bounding boxes and relation labels.Five annotators independently labeled each relation; inconsistent samples were revisited for consensus.
- 3.2. Social Relation Dataset: Fleiss’ kappa averages 0.62 across the eight relation traits, indicating substantial inter-rater agreement.
- 3.3. Baseline Method: The baseline Siamese-like DCN maps raw pixels from two detected faces to a shared representation for eight binary relation-classification tasks.The network uses tied branches, concatenated 2048-dimensional face features, and a 4096×256 projection into the shared representation.
- 3.3. Baseline Method: Adding face positions, relative positions, and scale ratios supplies spatial cues that are concatenated with the shared representation for relation prediction.
- 3.4. A Cross-Dataset Approach: Attribute-driven DCN pretraining captures expression, head pose, gender, and age despite missing labels across heterogeneous datasets.A correspondence-based bridging layer uses aligned facial-part associations as additional input, bringing representations from datasets with similar correspondence descriptors closer.
4. Experiments
Experiments evaluate social-relation prediction, cross-dataset facial-attribute learning, qualitative behavior, and video profiling. The proposed approach benefits from heterogeneous attribute pre-training and bridging-layer design, while spatial cues help selectively.
- Experimental setup: The relation dataset uses manually labeled binary attributes and mutually distinct training and testing partitions, including a movie-based subset intended to reduce annotator subjectivity.The facial-attribute resources comprise AFLW, CelebFaces, and Kaggle expression data; testing uses 2,000 faces each from AFLW and CelebFaces and 7,178 Kaggle faces.
- Social relation trait prediction: All deep-model variants outperform HOG+SVM, while expression pre-training raises overall performance from 64.0% to 70.6%.Pose pre-training is the next most influential attribute group, and spatial cues are useful overall but degrade some individual traits.
- Qualitative results: Qualitative examples produce positive trusting, warm, and friendly predictions for Obama with family members, while false positives are mainly associated with large face occlusions.The examples also show relation differences across contexts and a low trusting probability in an image associated with the US spying scandal.
- Further analyses: On Kaggle facial-expression recognition, the method achieves 75.10% accuracy versus the Kaggle winning method’s 71.2%.The result comes from fusing multiple sources through the proposed bridging layer.
- Bridging-layer analysis: The bridging layer improves recognition of most attributes, especially expressions, and performs better when used as input rather than output.Its face descriptor also groups samples from different datasets into coherent clusters with similar face patterns.
- Video application: In Iron Man, averaged predictions across five neighboring frames capture friendly conversation and competitive conflict between Tony Stark and Pepper Potts.The movie application profiles relation traits frame by frame after detecting faces and selecting frames containing both roles.
5. Conclusion
The paper establishes face-based social relation trait prediction as a challenging problem requiring complex facial-attribute recognition. It identifies bridging-layer deep models as essential for learning from multiple datasets with missing attribute labels and outlines broader future extensions.
- Face-based social relation prediction is challenging because accurate prediction relies on recognizing complex facial attributes.
- A deep model with a bridging layer is essential for exploiting multiple datasets that may have missing attribute labels.
- Future work will combine facial cues with environment context and body gesture for relation prediction.
- The framework could be extended to relations among more than two people using voting or graphical models.