PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Multimodal Interaction Reasoning for Dyadic Relationship Recognition

Wang Tang, Fethiye Irmak Dogan, Linbo Qing, Ce Zhu, Hatice Gunes

arXiv:2610.36249v1cs.CVcs.MM

TL;DR

Dyadic relationship recognition seeks to infer the latent relationship between two people from multimodal interaction, but existing approaches often leave relationship evidence implicit. HiRe-DRR organizes evidence by feature level, interaction type, and relationship hierarchy, using specialized interaction routing and conditional supervision. Across three benchmarks, it reports consistent improvements over representative image-, video-, and multimodal methods, while routing analyses indicate relationship-dependent interaction patterns.

  • Problem

    Existing approaches leave unclear which interaction patterns distinguish dyadic relationships and how evidence is structured across relationship categories.

    Source: B. Spatio-temporal Multimodal SRR · p. 3

  • Method

    HiRe-DRR organizes multimodal evidence across feature levels, cross-participant interaction types, and relationship hierarchy, with sparse interaction routing and conditional hierarchical supervision.

    Source: §I · p. 2

  • Results

    Experiments on three benchmarks show consistent improvements over representative image-, video-, and multimodal methods.

    Source: §VI · p. 12

  • Takeaways & Limitations

    Routing analyses indicate that dyadic relationships can be understood through relationship-dependent interaction evidence rather than uniformly aggregated multimodal cues.

    Source: §VI · p. 12

Abstract

from arXiv · show

Dyadic relationship recognition aims to identify the latent social relationship between two individuals from their multimodal interactions. Existing methods typically project heterogeneous cues into a shared representation and optimise a flat classification objective, overlooking that relationships may rely on different interaction patterns. For instance, distant relationships can often be inferred from direct behavioural correspondence, whereas close relationships are better reflected by coordinated affective, behavioural, and multimodal responses. To address this limitation, we propose HiRe-DRR, a Hierarchical Multimodal Interaction Reasoning framework that organises relationship evidence from three complementary perspectives: feature level, interaction type, and relationship hierarchy. HiRe-DRR decomposes facial, bodily, acoustic, and textual cues into low- and high-level representations and constructs homogeneous and heterogeneous cross-participant interactions. We further introduce the Mixture-of-Interaction Transformer (MoI-Former), which treats diverse interactions as specialised experts and employs relation-aware sparse routing to identify relationship-relevant evidence while preserving contextual interaction information. The proposed Hierarchical Conditional Loss (HC-Loss) exploits the hierarchical nature of social relationships through coarse-to-fine supervision, enabling structured relationship reasoning. Extensive experiments on three dyadic relationship benchmarks demonstrate that HiRe-DRR outperforms recent representative image-, video-, and multimodal methods. Our routing analysis further reveals that distant relationships tend to rely on low-level homogeneous interactions, whereas closer relationships focus on high-level heterogeneous interactions. Our code is publicly available at: https://github.com/tw-repository/HiRe-DRR.

I. INTRODUCTION

HiRe-DRR addresses the open question of how multimodal interaction evidence at different semantic levels and across relationship classes characterizes distinct dyadic relationships. It organizes evidence by feature level, interaction type, and relationship hierarchy, responding to limits in holistic fusion and flat classification.

I. INTRODUCTION

  • Low-level cues describe observable behavioral consistency, while high-level cues describe richer social semantics such as affect, engagement, and coordination.The introduction warns that projecting these levels into one space may obscure how each contributes to a relationship.Source: §I · p. 2
  • Homogeneous interactions link the same modalities across participants, whereas heterogeneous interactions connect different modalities and can reflect cross-modal responses.Uniform aggregation may conceal relationship-specific interaction patterns.Source: §I · p. 2
  • HiRe-DRR jointly models feature level, cross-participant interaction type, and relationship hierarchy for dyadic relationship recognition.It constructs homogeneous and heterogeneous interactions and applies hierarchical supervision.Source: §I · p. 2
  • MoI-Former treats interaction cues as specialized relationship experts and uses Top-K sparse routing to select and aggregate relevant evidence.Its routing preferences offer a view of how relationships depend on different feature levels and interaction types.Source: §I · p. 2
  • HC-Loss incorporates relationship hierarchy through binary conditional decisions, rather than treating all classes as independent targets.The motivating hierarchy proceeds from acquaintance status to friendship and then Very Good Friend status.Source: §I · p. 2

A. Static Image-based SRR

  • Static image-based SRR uses appearance, facial attributes, pairwise geometry, and scene context, but cannot capture recurring behavior or evolving participant responses.Later image methods add contextual semantic reasoning, including socially informative objects and activities.Source: A. Static Image-based SRR · pp. 2–3

B. Spatio-temporal Multimodal SRR

Video SRR has developed toward structured multimodal temporal reasoning, but many approaches remain character-centric and tailored to movie narratives. HiRe-DRR instead organizes real-world dyadic evidence through staged feature extraction, interaction construction, relationship reasoning, and hierarchical supervision.

B. Spatio-temporal Multimodal SRR

  • Video SRR has shifted toward structured multimodal temporal reasoning, yet many methods remain character-centric and tailored to movie narratives.For real-world dyadic interactions, AsyReC models asymmetric participant dependencies, while multimodal foundation-model work explores social reasoning and familiarity inference.Source: B. Spatio-temporal Multimodal SRR · p. 3
  • Existing approaches leave relationship evidence largely implicit, making it unclear which interaction patterns distinguish relationships or how evidence is structured across categories.HiRe-DRR addresses this gap by explicitly organizing evidence according to semantic characteristics, interaction structures, and relational hierarchy.Source: B. Spatio-temporal Multimodal SRR · p. 3

III. METHOD

  • HiRe-DRR partitions videos into synchronized face, body, audio, and text streams, extracts low- and high-level features, and constructs homogeneous and heterogeneous interactions.MoI-Former reasons over those interactions before HC-Loss applies hierarchy-aware conditional decisions.Source: §III · p. 3

1) Facial Feature Extraction:

  • The facial stream uses geometric cues such as landmarks and head pose for low-level features, and action units, gaze, valence-arousal, and appearance for high-level features.Frame-wise facial descriptors are aggregated into clip-level representations using statistical temporal pooling.Source: 1) Facial Feature Extraction · p. 3

L, H, U, G, AVA

The feature representations distinguish geometric body configuration from more expressive postural and appearance cues. Frame-wise body descriptors are pooled into clip-level representations and organized by semantic level.

2) Body Feature Extraction:

  • Body openness and orientation cues are derived because keypoints describe geometric configuration but do not explicitly capture expressive postural states.Openness measurements are normalized to reduce variation from body scale and camera distance.Source: 2) Body Feature Extraction · p. 4
  • Body openness characterizes spatial expansion of the torso and limbs, using torso length as a scale reference.The openness descriptor includes limb and joint distances and projected torso area, with measurements normalized by body scale.Source: 2) Body Feature Extraction · p. 4
  • Body orientation is represented using relative directions of the shoulder, hip, and torso axes.Their concatenation forms the orientation descriptor.Source: 2) Body Feature Extraction · p. 4
  • The low-level body representation uses pooled keypoints, while the high-level representation projects openness, orientation, and appearance cues.Frame-wise body descriptors are aggregated through statistical temporal pooling.Source: 2) Body Feature Extraction · p. 4

3) Audio Feature Extraction:

The audio stream separates low-level acoustic descriptors from higher-level vocal and temporal cues. Corresponding cues are projected into low- and high-level audio representations.

3) Audio Feature Extraction:

  • The low-level audio encoder extracts MFCC, fundamental frequency, and energy as spectral and prosodic descriptors.These descriptors capture complementary acoustic characteristics of speech.Source: 3) Audio Feature Extraction · p. 4
  • The high-level audio encoder extracts acoustic emotion, speech dynamics, and spectral voice characteristics.Speech activity and pauses describe temporal dynamics, while the spectral descriptor captures voice-quality variation.Source: 3) Audio Feature Extraction · pp. 4–5
  • The low- and high-level audio representations are obtained by projecting their corresponding acoustic cues.The audio projection functions are denoted audio(·).Source: 3) Audio Feature Extraction · p. 5

4) Text Feature Extraction:

Text features distinguish token-level linguistic representations from high-level textual sentiment. The section also describes constructing homogeneous and heterogeneous interactions across participants at each feature level.

4) Text Feature Extraction:

  • The low-level text encoder captures token-level linguistic representations, while the high-level encoder extracts textual sentiment.Source: 4) Text Feature Extraction · p. 5

B. Interaction Feature Construction

  • At each feature level, the method models both same-modality homogeneous interactions and cross-modality heterogeneous interactions between participants.The interaction construction pairs face, body, audio, and text representations into a 4 × 4 matrix.Source: B. Interaction Feature Construction · p. 5, Figure 3 · p. 5
  • Homogeneous interactions encode behavioral consistency and individual differences, whereas heterogeneous interactions use a bidirectional unit to model cross-modal dependencies while preserving modality-specific perspectives.Source: B. Interaction Feature Construction · p. 5

C. Mixture-of-Interaction Transformer

MoI-Former organizes interaction tokens as specialized experts and uses relation-aware sparse routing to select evidence for relationship reasoning. It retains the full interaction bank as contextual memory while selected tokens are refined and aggregated.

C. Mixture-of-Interaction Transformer

D. Relationship Classification with HC-Loss

HC-Loss represents relationship classes as a hierarchy and supervises conditional decisions along each sample’s path. The final objective combines this hierarchy-aware loss with standard classification loss.

D. Relationship Classification with HC-Loss

IV. EVALUATION SETUP

The evaluation uses three relationship-recognition benchmarks, with clip-level features and a non-overlapping dyadic cross-validation protocol. It reports class-wise recall, UAR, and mAP.

A. Datasets

  • NoXi includes 168 videos across four relationship classes with two participant perspectives, while UDIVA contains 1,160 videos for binary known-versus-unknown recognition.Source: A. Datasets · p. 7
  • SeamInt contains over 4,000 hours and 4,000 participants; experiments use a random sample of 2,738 videos spanning five relationship classes.Source: A. Datasets · p. 7

1) Feature Extraction:

  • Frozen modality-specific encoders extract complementary low- and high-level facial, bodily, acoustic, and textual features, while projection and interaction reasoning modules are optimized end-to-end.Source: 1) Feature Extraction · p. 7

2) Experimental Setup:

  • Experiments use three-fold cross-validation with non-overlapping dyadic video partitions and train for 100 epochs using AdamW.The setup uses batch size 64 and decays the initial learning rate by a factor of 10 every 15 epochs.Source: 2) Experimental Setup · p. 7

3) Evaluation Metrics:

  • Evaluation reports class-wise Recall, Unweighted Average Recall (UAR), and mean Average Precision (mAP) to assess class performance, class-balanced performance, and confidence ranking.Source: 3) Evaluation Metrics · p. 7

C. Baseline Models

HiRe-DRR is compared with representative image-, video-, and multimodal relationship-recognition methods, and achieves the best overall UAR/mUAR and mAP on all three benchmarks. The section also reports benchmark-specific performance and describes the baseline methods used.

C. Baseline Models

  • Image-based baselines include graph reasoning methods GR2N and HGM, while video-based PGCN uses pyramid graph convolution.Image-based methods are adapted by temporally aggregating frame-level visual features, without audio or text.Source: C. Baseline Models · p. 7
  • Multimodal baselines include visual-textual LIReC, cumulative-interaction CT, and facial, bodily, acoustic, and textual AsyReC.CT progressively accumulates historical interaction representations; AsyReC models directional interpersonal dependencies and temporal patterns.Source: C. Baseline Models · p. 7

A. Comparative Analyses

  • HiRe-DRR achieves the best overall UAR/mUAR and mAP among representative methods on all three dyadic relationship benchmarks.The comparison includes image-, video-, and multimodal methods.Source: A. Comparative Analyses · p. 7, Table II · p. 8
  • On NoXi, HiRe-DRR achieves 63.6% UAR on NoXi-I, 53.5% UAR on NoXi-J, 58.5% mUAR, and 63.1% mAP.It surpasses AsyReC by 10.3 and 11.0 percentage points in mUAR and mAP, respectively.Source: A. Comparative Analyses · pp. 7–8
  • On UDIVA, HiRe-DRR achieves 62.5% UAR and 66.8% mAP, exceeding AsyReC by 3.2 percentage points on both metrics.Known and Unknown recalls are 60.4% and 64.8%, respectively; the paper attributes smaller gains partly to UDIVA’s seated, upper-body setting.Source: A. Comparative Analyses · p. 8
  • On SeamInt, HiRe-DRR obtains 63.0% UAR and 68.5% mAP, outperforming AsyReC by 12.7 and 16.3 percentage points, respectively.The paper links the larger gains to SeamInt’s broader relationship taxonomy and richer behavioural variations.Source: A. Comparative Analyses · p. 8

B. Ablation Analysis

The ablations assess the contributions of feature levels, interaction reasoning, and hierarchical supervision. Routing analyses show that selected evidence varies with relationship category, while token masking tests the recognition value of routed interactions.

1) Ablation of the Entire Framework:

  • Adding HC-Loss to low- and high-level interaction reasoning yields the best results on all benchmarks, with larger gains on NoXi and SeamInt than on binary-label UDIVA.Combining both feature levels with MoI-Former improves performance from 49.0% to 55.1% on NoXi, 55.9% to 65.9% on UDIVA, and 55.3% to 62.4% on SeamInt.Source: 1) Ablation of the Entire Framework · p. 9
  • Across modalities, performance improves through L, H, LH+MoI, and LH+MoI+HC configurations, supporting complementary roles for feature levels, interaction reasoning, and hierarchical supervision.NoXi face results rise from 39.7 to 45.8, 51.3, and 61.5; SeamInt audio results rise from 37.5 to 49.1, 56.5, and 64.5.Source: 1) Ablation of the Entire Framework · p. 9

2) Ablation of the Feature Level:

  • Across NoXi and SeamInt, routing shifts toward high-level evidence for socially closer relationships, while both feature levels remain active.Low-level cues capture observable behavioural correspondence; high-level cues provide affective and semantic evidence for fine-grained understanding.Source: 2) Ablation of the Feature Level · p. 9, Figure 5 · p. 9

3) Ablation of the Interaction Type:

  • Routing shifts toward heterogeneous interactions for closer relationships, while homogeneous interactions retain preference across categories.Homogeneous interactions link the same modalities across participants; heterogeneous interactions capture cross-modal responses.Source: 3) Ablation of the Interaction Type · pp. 9–10, Figure 6 · p. 9
  • NoXi routing favors low-level F↔F and T↔T, while closer Very Good Friend cases increasingly activate high-level A↔F and A↔B interactions.SeamInt shows heterogeneous low-level pathways and, for Family and Couple, high-level B↔B, F↔B, and B↔T interactions.Source: 3) Ablation of the Interaction Type · p. 10, Figure 7 · p. 10
  • Fig. 8 presents t-SNE projections of relationship representations learned with and without HC-Loss on NoXi and SeamInt.The caption identifies the comparison but does not state its outcome.Source: Figure 8 · p. 10
  • Masking the five high-preference tokens consistently reduces mAP, whereas masking the five low-preference tokens causes no degradation and can slightly improve results.The authors interpret low-preference tokens as potentially redundant.Source: 3) Ablation of the Interaction Type · pp. 10–11, Table V · p. 10
  • Qualitative cases show relationship-dependent Top-5 interaction selection, including correct and failure cases under ambiguous relationship patterns.Fig. 9 uses blue and orange for low- and high-level interactions, with colour intensity indicating routing preference.Source: Figure 9 · p. 11

4) Ablation of the HC-Loss:

HiRe-DRR improves on representative methods across three benchmarks, and its analyses show relationship-specific interaction patterns. The HC-Loss ablation and qualitative cases illustrate both clearer class representations and the challenge of overlapping evidence.

4) Ablation of the HC-Loss:

  • With HC-Loss, relationship representations become more compact and discriminative, with clearer class boundaries than without hierarchical supervision.Without HC-Loss, representations show substantial inter-class overlap and dispersed distributions; HC-Loss particularly enhances discrimination between semantically adjacent classes.Source: 4) Ablation of the HC-Loss · p. 11

C. Qualitative Analyses

  • Successful cases show MoI-Former selecting interactions aligned with the relationship: high-level affective cues for Family and low-level observable correspondences for Stranger.Couple recognition also uses structured facial, bodily, and cross-modal interactions beyond observable motion patterns.Source: C. Qualitative Analyses · p. 11
  • Shared interaction patterns can confuse relationship categories: low-level cues may not distinguish Family from Acquaintance, while high-level responses may lead to confusing Acquaintance with Friend.The cases indicate that cues associated with one relationship can also appear in another.Source: C. Qualitative Analyses · pp. 11–12

VI. DISCUSSION AND CONCLUSION

  • Relationship-specific interaction patterns may provide context for personalized, socially aware responses in human-robot interaction and social assistance.The discussion also identifies possible relevance to interactive agents, human activity understanding, and social interaction analysis.Source: §VI · p. 12
  • Across three benchmarks, HiRe-DRR consistently improves over representative image-, video-, and multimodal methods, with distant relationships relying more on observable correspondences and closer ones on high-level semantic and heterogeneous interactions.Source: §VI · p. 12
  • Current benchmarks mainly capture short-term interactions, while real-world relationships evolve over longer periods and across changing contexts and broader social structures.Future work includes long-range temporal modelling, multi-person relational reasoning, and more adaptive evidence discovery.Source: §VI · p. 12

Related papers

Loading 2610.36249v1…