Source-linked AI summary
From Facial Expression Recognition to Interpersonal Relation Prediction
Zhanpeng Zhang, Ping Luo, Chen Change Loy, Xiaoou Tang
TL;DR
The paper asks whether fine-grained interpersonal traits can be characterized and quantified from face images in the wild. It combines multitask facial-attribute learning with attribute propagation across heterogeneous datasets, then uses the learned network in a Siamese model. Experiments report effective facial expression recognition and accurate fine-grained interpersonal relation prediction.
Problem
The paper investigates fine-grained interpersonal relation traits from face images beyond single-person facial expression recognition, despite incomplete labels and heterogeneous attribute datasets.
Method
It learns expression, gender, age, and pose jointly, propagates missing attributes across heterogeneous datasets with a graphical-model approach, and initializes a Siamese-like face-pair relation model.
Results
Extensive experiments demonstrate effectiveness on facial expression recognition and interpersonal relation prediction, including accurate fine-grained relation prediction from mutual face context.
Takeaways & Limitations
The learned facial representation provides a foundation for recognizing interpersonal relations from raw face-image pairs.
Takeaways & Limitations
Relation prediction is constrained by inconsistent face scale and location across image sources, and dominance depends on social role and environmental context.
Abstract
from arXiv · showhide
Interpersonal relation defines the association, e.g., warm, friendliness, and dominance, between two or more people. Motivated by psychological studies, we investigate if such fine-grained and high-level relation traits can be characterized and quantified from face images in the wild. We address this challenging problem by first studying a deep network architecture for robust recognition of facial expressions. Unlike existing models that typically learn from facial expression labels alone, we devise an effective multitask network that is capable of learning from rich auxiliary attributes such as gender, age, and head pose, beyond just facial expression data. While conventional supervised training requires datasets with complete labels (e.g., all samples must be labeled with gender, age, and expression), we show that this requirement can be relaxed via a novel attribute propagation method. The approach further allows us to leverage the inherent correspondences between heterogeneous attribute sources despite the disparate distributions of different datasets. With the network we demonstrate state-of-the-art results on existing facial expression recognition benchmarks. To predict inter-personal relation, we use the expression recognition network as branches for a Siamese model. Extensive experiments show that our model is capable of mining mutual context of faces for accurate fine-grained interpersonal prediction.
1 Introduction
The paper extends facial expression recognition toward fine-grained interpersonal relation prediction from face pairs. It combines mutual face context, multitask attribute learning, and attribute propagation across heterogeneous datasets.
- Motivation: Interpersonal relation prediction targets traits such as friendliness, warmth, and dominance from faces co-existing in an image.The task goes beyond analyzing facial motion and feature changes in a single subject.
- Challenges: Existing methods largely analyze one face, while interpersonal relations require joint reasoning over pairwise faces and multiple attributes.Relevant attributes include expression, gender, age, and head pose.
- Approach: A Siamese-like network uses two shared-parameter convolutional branches whose fused outputs support joint relation reasoning.Each branch processes one of the faces, with each face serving as mutual context for the other.
- Approach: The multitask framework learns facial expression representations alongside auxiliary attributes including head pose, gender, and age.The study gathers multiple expression and attribute datasets and adds the large-scale ExpW dataset, formed from over 90,000 web images.
- Approach: MRF-based attribute propagation infers missing labels during training and bridges heterogeneous datasets with different annotations and statistical distributions.The method conditions inferred labels on appearance similarity with annotated samples.
- Contributions: The study reports state-of-the-art facial expression benchmark results and accurate fine-grained interpersonal prediction from mutual face context.It also introduces a new interpersonal relation dataset labeled with pairwise relation traits.
2 Related Work
Prior work covers facial expression recognition, face analysis, and human interaction, but generally does not perform fine-grained interpersonal relation inference from jointly considered faces. The paper positions its approach as an appearance-based, multi-attribute model for abstract relation traits.
- Social Signal Processing: Social signal processing includes facial expression recognition and broader efforts to understand interpersonal relations from visual data.The paper situates interpersonal relation understanding within this multidisciplinary area.
- Facial Expression Recognition: Facial expression methods use either facial action or appearance-based representations for expression classification.Action-based methods exploit geometry or action units, while appearance-based methods extract features from patches or whole faces.
- Facial Expression Recognition: Most existing appearance-based approaches focus on a single person and prototypical expressions, limiting direct use for interpersonal relation inference.Interpersonal relations involve additional factors such as age and gender.
- Human Interaction and Group Behavior: Human interaction studies often identify social groups, roles, or coarse interaction classes rather than fine-grained relation traits between individuals.Other group-behavior work emphasizes actions such as hugging, handshaking, or walking.
- Human Interaction and Group Behavior: This work differs by recognizing abstract, high-level interpersonal relation traits directly from faces rather than relying mainly on spatial configuration, actions, or detected visual concepts.The approach emphasizes face information for relations between individuals.
- Datasets: Existing facial expression datasets are often controlled or too small for robust deep-network training, motivating comparison with the proposed ExpW dataset.Table 1 compares popular facial expression datasets with ExpW.
3 Face Expression and Interpersonal Relation Datasets
The paper introduces ExpW and a new interpersonal-relation dataset, defining relation traits from Kiesler’s interpersonal circle and annotating them in diverse face-pair images.
- Expression in-the-Wild Dataset: ExpW contains 91,793 manually labeled faces collected from web images and annotated with seven basic expression categories.The collection used emotion-related keyword queries, face detection, and manual expression labeling; non-face images were removed.
- Interpersonal Relation Dataset: The interpersonal-relation dataset contains 8,016 web and movie images with face bounding boxes and pairwise relation labels.Its images include substantial variation in pose, occlusion, and illumination, and come from politicians’ news photos, social media, and movies.
- Interpersonal Relation Dataset: Kiesler’s 16 interpersonal-circle segments are grouped into eight binary relation traits for annotating face pairs.The traits include opposing relation directions, such as friendly and hostile, and are defined using psychological studies.
- Interpersonal Relation Dataset: Relation traits are non-exclusive, so a face pair may receive more than one relation label when semantically close traits co-occur.This allows overlapping traits such as friendly and trusting rather than forcefully suppressing one during prediction.
- Interpersonal Relation Dataset: Five annotators independently labeled each relation, with consensus procedures for inconsistent cases and an average Fleiss’ kappa of 0.62.Annotators used related adjectives, example images, movie plots and subtitles, and measurable rules to support labeling.
4 Facial Expression and Attributes Recognition
The paper learns a rich face representation from heterogeneous datasets with incomplete attribute annotations by combining multitask network training and alternating MRF-based attribute propagation. This framework uses appearance similarity and attribute correlations to infer missing labels, refine the network, and support facial expression and attribute recognition.
- Representation learning: Jointly predicting facial expression and auxiliary attributes produces a representation intended to capture richer facial information for relation prediction.The auxiliary attributes include gender, age, and head pose.
- Problem: Missing and non-overlapping attribute labels across face datasets limit supervision and make it difficult to exploit each image fully.AFLW contains gender and pose labels, whereas ExpW contains expressions; the resulting sparse supervision can also hinder convergence.
- Framework: The proposed two-stage framework learns a DCN for the union of attributes across heterogeneous datasets.Stage 1 initializes the network with available labels while masking errors for missing attributes; Stage 2 alternates propagation and representation learning.
- Alternating optimization: The second training stage refines the DCN using both ground-truth labels and MRF-generated pseudo labels.The process alternates between generating pseudo labels and minimizing attribute-classification error.
- Attribute propagation: Attribute propagation infers missing labels from face-appearance similarity and correlations among attributes within an MRF.The method uses face representations as observed variables and models attribute co-occurrence through pairwise relations.
5 Interpersonal Relation Prediction from Face Images
The interpersonal-relation model places two identical expression-recognition DCNs in a Siamese architecture and fuses their features with spatial cues. It predicts eight binary relation traits from the shared representation, with experiments supporting accurate fine-grained prediction.
- Outcome: Experiments show that the model mines mutual context between faces for accurate fine-grained interpersonal prediction.The two branches are fused so that each face contributes context for reasoning about the other.
- Architecture: The model uses two identical DCNs as Siamese branches to extract high-level features from a pair of face images.The branches share the expression-recognition architecture and are initialized from the learned face representation.
- Shared representation: Concatenated face features are projected into a shared 256-dimensional representation for relation-trait prediction.Each face feature is 1024-dimensional, and the shared representation supports eight binary classification tasks.
- Spatial cues: Spatial cues describing face positions, bounding-box sizes, and relative scale are combined with the shared face representation.These cues provide positional and scale information for joint relation reasoning.
- Prediction: Each relation trait is modeled as a binary variable whose probability is predicted from the shared representation and spatial features.The model uses sigmoid probabilities and linear classifiers, with tied branch filters.
- Training: The resulting network is trained by minimizing cross-entropy terms with parameter weight decay using stochastic gradient descent.The branch parameters are initialized from the face representation learned in the earlier stage.
6 Experiments
The experiments evaluate the proposed networks on facial expression, attribute, and interpersonal-relation tasks. Results show benefits from attribute propagation, cross-dataset pre-training, and learned face-pair representations, while context and spatial-cue limitations remain.
- Experimental setup: The experiments evaluate a base DCN on facial expression and attribute recognition and a full Siamese framework on interpersonal relation prediction.The combined AFLW, CelebA, and ExpW evaluation uses 318,778 face images, including 5,400 test images.
- Facial expression and attribute recognition: Attribute propagation generally improves balanced accuracy, especially for attributes with rare positive samples such as narrow eyes and goatee.The authors conjecture that propagation leverages initially unannotated samples from multiple datasets.
- Facial expression and attribute recognition: Alternating attribute propagation and representation learning yields gradually improved average balanced accuracy across iterations.The method also automatically annotates initially unlabeled positive attribute samples, including challenging unconstrained poses and expressions.
- Facial expression recognition: On SFEW, the approach with and without fine-tuning outperforms state-of-the-art methods, while most errors arise from ambiguous cases.The evaluation uses overall classification accuracy, following the convention of current studies.
- Facial expression recognition: On CK+, the approach achieves better results than compared state-of-the-art methods despite near-saturated performance on the dataset.The comparison reports averaged accuracy over the six basic expressions.
- Interpersonal relation prediction: Expression pre-training improves interpersonal-relation performance from 64.1% to 67.7%, while pose pre-training provides the next-largest influence.All proposed S-DCN variants outperform the HOG+SVM baseline; the authors associate pose relevance with people looking toward one another or in the same direction.
- Interpersonal relation prediction: The S-DCN shows reliable relation prediction on a 522-frame movie subset evaluated across eight relation traits.This subset was selected to reduce subjective effects in annotation by providing annotators with richer auxiliary cues.
- Interpersonal relation prediction: False positives are partly caused by missing context, and large occlusions also produce failures.A serious conversation was classified as competing because the model could not access the characters’ book-reading context.
7 Conclusion
The paper studies interpersonal-relation prediction from face images through a two-stage system. It combines attribute-aware single-face representation learning with a Siamese-like pairwise model, and experiments demonstrate effectiveness on both expression recognition and relation prediction.
- Conclusion: The paper studies the challenging problem of predicting interpersonal relation from face images.The target includes fine-grained relation traits beyond single-person facial-expression recognition.
- Conclusion: A deep convolutional network learns facial expression and rich attributes while attribute propagation addresses heterogeneous sources with missing labels.The propagation uses correspondences among heterogeneous sources through a graphical model.
- Conclusion: A Siamese-like framework maps raw pixels from face pairs to relation traits after initialization from the single-face network.The framework is trained for interpersonal-relation prediction.
- Conclusion: Extensive experiments demonstrate the effectiveness of the proposed methods on facial expression recognition and interpersonal relation prediction.
- Conclusion: Future work will combine face-based relation traits with body-driven immediacy cues for more accurate interpersonal relation prediction.