A Domain Based Approach to Social Relation Recognition

1. Introduction

• Prior computational models often cover only a handful of ad hoc social-relation classes despite the diversity and complexity of social life.Visual understanding of social relations is motivated in part by socially appropriate assistants and informing users about potential privacy risks.Source: §1 · pp. 1–2
• The dataset extends PIPA with 26,915 person-pair annotations and hierarchical labels for five social domains and their relations.The domain framework is Bugental’s theory.Source: §1 · p. 2
• The paper adopts a holistic yet computationally concrete conceptualization of social life from social psychology.It also collects semantic attributes from human body and head images based on the social psychology study.Source: §1 · p. 2
• Semantic attributes perform best on the dataset and yield interpretable concepts that correlate with predictions from social psychology theory.The paper also presents the first models to predict social domains and relations from images.Source: §1 · p. 2

2. Related works

• Kinship studies focus on familial relations and use visual patterns, facial features, attributes, and landmarks; examples include spouse assumptions based on age and gender.Prior work also studies optimal face features for parent-offspring resemblance.Source: §2 · p. 2
• Event-based social-role recognition models expected behaviors such as birthday-party roles, while this paper targets identity-specified interpersonal relations that imply more permanent roles.Occupation recognition can combine personal attributes with semantic context, such as a waiter standing beside seated consumers in a restaurant.Source: §2 · p. 2
• Other work studies social categories and occupations, intimacy from interactive poses, or relation traits such as warmth and dominance.The paper distinguishes its goal as categorizing relations across social domains.Source: §2 · pp. 2–3

3. Social psychology theories

• The paper seeks a social-psychology theory with broad coverage, concrete relations, and concepts that can be modeled from images and video.It identifies Bugental’s social domain theory as a suitable candidate.Source: §3 · p. 3

3.1. Social domain theory [1]

• Bugental’s theory partitions social life into five domains whose concrete behaviors and cues can be recognized visually and used to derive relation labels.Examples include protective kinship cues in Attachment, mutual activities in Reciprocity, and group clothing or joint activities in Coalitional groups.Source: §3.1 · p. 3

3.2. Related theories

• Alternative theories are less suitable because some are abstract, some concern resources difficult to infer visually, and Fiske’s theory emphasizes individual experience over visible behavior and environments.Bugental’s theory offers numerous social cues that can inform computational models.Source: §3.2 · p. 3

4. Social domain and relation data set

• The study starts from PIPA, whose Flickr photos span social situations and include person identities and head bounding boxes for selecting pairs and defining data splits.PIPA contains 37,107 photos, 63,188 person instances, and 2,356 identities.Source: §4 · pp. 3–4

4.1. Social relation labels

• The final annotation label set contains 16 social relations, derived from domain examples, expanded through pre-annotation, and filtered for insufficient occurrences or agreement.Five annotators labeled all PIPA person pairs using an initial list of 20 relations.Source: §4.1 · p. 4

4.2. Annotation process

• Five annotators labeled 26,915 person pairs, producing 134,556 annotations; about 8% were marked “maybe” and retained in the statistics.Annotators could assign up to three labels per pair or skip it.Source: §4.3 · p. 4, §4.2 · p. 4

4.3. Label statistics

• For 53% of person pairs, at most one relation was chosen across annotators; four relations were chosen for 0.8% of pairs.The paper describes highly ambiguous cases as less prominent than expected and argues that visual recognition is feasible.Source: §4.3 · p. 5

4.4. Consistency analysis

• 42% of person pairs have complete agreement (agr=5), and annotations with agr≥3—82.6% of the annotations—are used as ground truth.The agr≥3 threshold is called consistency=3.Source: §4.4 · p. 5

5.1. End-to-end model

• A double-stream CaffeNet maps paired head or body images to 5 social domains or 16 social relations.Its streams have independently trained convolutional weights; after five convolutional layers, their features are concatenated for classification.Source: §5.1 · p. 5, Figure 5 · p. 6

5.2. Semantic attributes

• The semantic-attribute approach predicts an intermediate attribute representation, then uses it to predict domains and relations.The paper motivates this representation through interpretability, links to social-domain theory, and the possibility of leveraging other attribute-labeled datasets.Source: §5.2 · p. 5
• Attributes cover age, gender, location and scale, head appearance, pose, emotion, clothing, proximity, and activity, using head and body regions where applicable.The paper uses existing datasets or published models to train or obtain many attribute features; its body region is defined as 3× head width and 6× head height.Source: §5 · p. 5, §5.2 · p. 6

6.1. Data splits

• All-class splits test recognition across represented classes, while single-relation splits withhold one relation to test domain prediction for unseen relations.The single-relation setup uses 15 train-validation-test groups; lovers/spouses remain in training when testing other relations because otherwise the Mating domain has no training data.Source: §6.1 · p. 7

6.2. Recognition

• 57.2% relation and 67.8% domain recognition were the best results; semantic-attribute models improved domain generalization by 14.8% over end-to-end finetuned.The best recognition results were 8.6% and 4.6% higher than the best baseline numbers, respectively; end-to-end finetuned reached 18.5% in domain generalization, around the 20% chance level.Source: Table 1 · p. 7, §6.2 · p. 7

6.3. Analysis of semantic attributes

• Most individual attributes contribute relatively more to domain than relation recognition, so the authors conclude that relation classification needs more attributes.They describe relation classification as more challenging because it has more classes and finer granularity.Source: §6.3 · p. 8
• Activity and clothing rank among the top two attribute categories for both tasks, while age and gender also contribute, with body gender and head age relatively important.Activity and clothing rank highly even though their models were transferred from other datasets.Source: §6.3 · p. 8
• Combined attributes correctly recognize some difficult examples, while unusual behavior, confused visual cues, or unclear body and head appearances accompany errors in others.The authors describe a case where activity and clothing cues dominate and lead to an incorrect prediction, despite age- or gender-finetuned models predicting correctly.Source: §6.3 · pp. 8–9

• Semantic attributes are important for recognizing social relations and support inspectable models connected to social-psychology theory.Source: §7 · p. 9
• The paper evaluates recognition of social relations and domains, as well as generalization of domain recognition across relations.Source: §7 · p. 9
• The authors envision empirical study of social relations that remains grounded and interpretable in social-psychology theories.Source: §7 · p. 9

Annotation tool

• Annotators independently labeled PIPA photos showing pairs of head boxes, choosing from 20 relation candidates; four labels were later filtered for low occurrence or agreement.Photos with one identity or more than five identities were unused; each remaining photo contained 1–10 person pairs.Source: Annotation tool · p. 11

Annotation samples

• Annotators received domain explanations and reference samples; Getty Images examples were mostly posed and helped clarify relations’ visual appearances.PIPA reference samples came from the 10% used when choosing the relation label list.Source: Annotation samples · p. 11

Supplementary results

• The all-attribute model correctly predicts positive examples that single-attribute models often misclassify; activity and clothing dominate father-child recognition.For one father-child example, all but the activity and clothing single-attribute models predicted incorrectly.Source: Supplementary results · p. 11
• Table 2 supplies accuracies for single-attribute and all-attribute models used to calculate the attribute coordinates shown in Figure 6.The accompanying example computes body-age coordinates by dividing its domain and relation accuracies by the corresponding all-attribute accuracies.Source: Supplementary results · p. 11, Table 2 · p. 11