PaperlayerView exact revision on arXiv

Source-linked AI summary

Inter-X: Towards Versatile Human-Human Interaction Analysis

Liang Xu, Xintao Lv, Yichao Yan, Xin Jin, Shuwen Wu, Congsheng Xu, Yifan Liu, Yizhou Zhou, Fengyun Rao, Xingdong Sheng, Yunhui Liu, Wenjun Zeng, Xiaokang Yang

arXiv:2312.16051v1cs.CV

TL;DR

Existing human-human interaction datasets lack accurate body motion, detailed hand gestures, and fine-grained descriptions, limiting comprehensive coverage of interaction analysis. Inter-X combines motion capture with diverse annotations and a benchmark spanning perceptual and generative tasks. The reported results characterize the dataset as challenging for both directions of interaction analysis.

  • Problem

    Existing human-human interaction datasets lack comprehensive coverage, including high-fidelity finger motion, fine-grained descriptions, and interaction-order and social-context annotations.

    Source: §1 · p. 2

  • Method

    Inter-X combines optical body motion capture and inertial hand tracking with fine-grained text, action, order, relationship, and personality annotations for four categories of downstream tasks.

    Source: §1 · pp. 2–3

  • Results

    The reported qualitative and quantitative results show that Inter-X poses challenges for human-human interaction perceptual and generative tasks.

    Source: §7 · p. 8

  • Takeaways & Limitations

    The authors present Inter-X as a unified testing ground for studying versatile human-human interaction analysis across perceptual and generative tasks.

    Source: §1 · p. 2, §7 · p. 8

    The conclusion's stated applications are text-conditioned interaction synthesis for AR/VR and gaming, and motion-based action recognition for intelligent surveillance.

    Source: §7 · p. 9

Abstract

from arXiv · show

The analysis of the ubiquitous human-human interactions is pivotal for understanding humans as social beings. Existing human-human interaction datasets typically suffer from inaccurate body motions, lack of hand gestures and fine-grained textual descriptions. To better perceive and generate human-human interactions, we propose Inter-X, a currently largest human-human interaction dataset with accurate body movements and diverse interaction patterns, together with detailed hand gestures. The dataset includes ~11K interaction sequences and more than 8.1M frames. We also equip Inter-X with versatile annotations of more than 34K fine-grained human part-level textual descriptions, semantic interaction categories, interaction order, and the relationship and personality of the subjects. Based on the elaborate annotations, we propose a unified benchmark composed of 4 categories of downstream tasks from both the perceptual and generative directions. Extensive experiments and comprehensive analysis show that Inter-X serves as a testbed for promoting the development of versatile human-human interaction analysis. Our dataset and benchmark will be publicly available for research purposes.

1. Introduction

Human-human interaction analysis remains challenging because interactions are diverse and self-occluding, while existing datasets lack comprehensive coverage. Inter-X combines precise motion and detailed annotations with a benchmark spanning perceptual and generative tasks.

1. Introduction

  • Existing datasets lack high-fidelity finger motion, fine-grained part-level descriptions, actor–reactor order, and relationship and personality information.The paper links these gaps respectively to expressive interaction, controllable text-driven generation, interaction asymmetry, and social context.Source: §1 · p. 2
  • Inter-X contains about 11K sequences across 40 categories and more than 8.1M frames, captured with optical body motion capture and inertial hand tracking.The dataset includes 89 subjects with varied social relationships and collected familiarity levels and Big Five personalities.Source: §1 · p. 2
  • The unified benchmark covers four downstream-task categories, split between perceptual and generative tasks, and evaluates existing methods while supplying baselines and metrics for remaining tasks.Source: §1 · p. 2
  • Inter-X adds fine-grained human-part text descriptions, semantic action categories, causal interaction order, and relationship and personality annotations.Source: §1 · p. 3

2. Related work

Related work spans human-motion datasets, interaction datasets, perceptual analysis, generation, and multimodal data. Inter-X is positioned against these lines of work through its motion quality, annotation detail, and breadth of interaction-related tasks.

2.1. Human motion datasets

  • Human-motion datasets use action labels or text descriptions, while other datasets add audio or scene and object conditions for human-centric tasks.Source: §2.1 · p. 3

2.2. Human-human interaction datasets

  • Prior human-human interaction datasets vary in size, modality, and functionality; Inter-X claims advantages in motion quality, fine-grained text, hand gestures, and annotation breadth.The comparison particularly notes InterHuman as a large-scale interaction dataset with textual annotations.Source: §2.2 · p. 3

2.3. Perceptive tasks for human motion

  • Human interaction recognition is a subfield of skeleton-based action recognition that models semantic correlations between people, and motion can also carry biometric cues.Prior work uses movement for gait identification and personality prediction; Inter-X is presented as supporting action recognition and assessment of relationships and personalities.Source: §2.3 · p. 3

2.4. Generative tasks for human motion

  • Human-motion generation uses action labels, text, or audio, while interaction work also studies multi-person generation, reaction generation, and stylized motion.Inter-X's interaction-order annotations are presented as supporting reaction generation, while personality and relationship labels can serve as style factors.Source: §2.4 · p. 3

2.5. Multimodality in vision

  • Inter-X combines multiple modalities and unifies several downstream-task categories to support deeper understanding of human-human interactions.Source: §2.5 · p. 3

3. The Inter-X Dataset

Inter-X comprises 11,388 interaction sequences across 40 categories and 89 subjects. Its capture system combines optical body motion capture with inertial gloves, followed by alignment and segmentation of the recorded sequences.

3. The Inter-X Dataset

  • Inter-X contains 11,388 interaction sequences, more than 8.1M frames, 40 daily interaction categories, and 89 subjects.Source: §3 · p. 3

3.1. Data Capturing System

  • The capture system prioritizes accurate body motion with optical MoCap and uses inertial gloves to capture finger gestures despite occlusion.The body and hand systems are spatially integrated through a triangular marker bracket and temporally synchronized; capture sessions use timecodes, with five categories and five repetitions per batch.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Data Postprocessing

  • Postprocessing aligns body and hand sequences, then annotators mark the start and end frames of atomic interaction snippets for trimming.Temporal alignment uses the sequence intersection, while spatial integration uses the shared wrist rotation from the triangular locating bracket.Source: §3.2 · p. 4

4. Dataset Taxonomy

Inter-X combines detailed interaction motion data with annotations spanning actions, text, actor–reactor order, relationships, and personality. Its dataset taxonomy describes how these data and labels are collected and organized.

4. Dataset Taxonomy

  • Inter-X contains 13,888 SMPL-X motion-sequence pairs, 273,312 synthetic multi-view RGB videos, 34,164 detailed descriptions, and labels for 40 action categories, interaction order, relationships, and personality.The relationship labels cover 59 groups, and personality labels cover 89 volunteers.Source: §4 · p. 4

4.1. Interaction data

  • SMPL-X represents body pose, shape, and translation; captured keypoints guide parameter fitting, and Unreal Engine renders six-camera RGB videos across four scenes.The synthetic videos use 1920×1080 resolution at 30 fps.Source: §4.1 · p. 4

4.2. Action categories

  • The 40 daily interaction categories are performed with varied actions, reactions, and body states, with each interaction repeated five times.Examples include different ways of raising hands and varied responses to being pushed.Source: §4.2 · pp. 4–5

4.3. Text descriptions

  • Three annotators describe each sequence’s coarse body movements, finger movements, and relative orientations; descriptions average about 35 words.Collected descriptions are typo-corrected with GPT-3.5 and spot-checked.Source: §4.3 · p. 5

4.4. Interaction Order

  • Volunteers label which person acts and which reacts in each atomic interaction sequence.Source: §4.4 · p. 5

4.5. Relationship & Personality

  • Personality is measured with the Big Five inventory, while volunteers report familiarity and declare one of five social relationship categories.The relationship categories are strangers, friends, lovers, schoolmates, and family.Source: §4.5 · p. 5

5. Task Taxonomy

Inter-X’s annotations define a benchmark spanning generative and perceptual human-interaction tasks. The tasks cover text and action conditioning, actor–reactor order, and relationship- or personality-based customization and assessment.

5. Task Taxonomy

  • The taxonomy pairs interaction motion with action, text, causal order, relationship, and personality labels to define downstream tasks.Source: §5 · p. 5

5.1. Texts related Tasks

  • Text-related tasks include generating interactions from fine-grained descriptions and captioning interaction sequences instead of classifying their action category.The generation task targets subtle hand gestures and alignment between part-level text and motion.Source: §5.1 · p. 5

5.2. Actions related Tasks

  • Action-related tasks generate diverse interactions from action labels and recognize interactions, with fine hand movements proposed as useful recognition input.Source: §5.2 · pp. 5–6

5.3. Interaction-order related Tasks

  • Interaction-order tasks generate human reactions and infer which participant is the actor or reactor from a sequence.Source: §5.3 · p. 6

5.4. Relationship & Personality related Tasks

  • Relationship and personality labels support stylized interaction generation and assessment of personality and relationships from motion.The proposed applications include education, medicine, and sports.Source: §5.4 · p. 6

6. Experiments

Experiments evaluate Inter-X on text- and action-conditioned interaction generation, human reaction generation, and interaction recognition. Results identify leading methods across generation tasks, while recognition accuracy remains limited.

6. Experiments

  • The experiments cover four tasks in the main manuscript, using Inter-X motion data and task-specific evaluation protocols.For text-conditioned generation, the authors use an 0.8/0.15/0.05 train/test/validation split and evaluate latent distance, diversity, multimodality, retrieval precision, and text-motion distance.Source: §6 · p. 6, §6.1 · p. 6, §6.2 · p. 7

6.1. Text-conditioned Interaction Generation

  • InterGen [55] achieves state-of-the-art text-conditioned generation performance except on MM Dist, while ComMDM [75] has the worst R Precision scores.Inter-X generations show more detailed hand movements and greater precision than InterHuman generations, which the authors describe as implausible for the three illustrated actions.Source: §6.1 · p. 6, Table 2 · p. 7, Figure 4 · p. 7

6.2. Action-conditioned Interaction Generation

  • Actformer achieves the best FID and action recognition accuracy, MDM [82] the best Multimod. score, and MDM-GRU [20] [82] the best diversity score.The authors note substantial potential for further improvements.Source: §6.2 · p. 7, Table 3 · p. 8

6.3. Human Reaction Generation

  • AGRoL [25] yields the best performance across all reported human reaction generation metrics, while the GRU architecture achieves the worst results..Source: §6.3 · p. 8, Table 4 · p. 8

6.4. Human Interaction Recognition

  • MS-G3D achieves the best Top-1 recognition accuracy at 83.30%, which the authors describe as unsatisfactory.They suggest dexterous hand gestures and action/reaction diversity may pose new challenges and opportunities.Source: §6.4 · p. 8, Table 5 · p. 8

7. Conclusion and Limitation

Inter-X combines precise, diverse interaction motion and hand gestures with annotations for four categories of downstream tasks. Its scope excludes facial expressions and long interaction sequences.

7. Conclusion and Limitation

  • Inter-X includes high-precision body motion, diverse interactions, subtle hand gestures, and annotations for text, semantic categories, interaction order, relationships, and personality.The authors report that qualitative and quantitative results show challenges for perceptual and generative interaction tasks.Source: §7 · p. 8
  • Facial expressions are absent because the dataset uses an indoor MoCap venue and non-professional actors, and the authors consider expression-motion correlation unreliable.They suggest natural outdoor scenes or professional actors as possible alternatives for exploring emotion and interaction.Source: §7 · p. 8
  • The 11,388 sequences are atomic rather than long interactions, so the dataset does not capture longer durations and frequent transitions described as common in real-world interactions..Source: §7 · p. 8

A. Extra experiments

The supplementary experiments include human interaction captioning. TM2T leads the reported captioning metrics, while the authors attribute weaker RAEs results to difficulty modeling long-term dependencies.

A. Extra experiments

  • The extra experiments cover four settings, including human interaction captioning, causal order inference, stylized interaction generation, and personality assessment..Source: A. Extra experiments · p. 13

A.1. Human interaction captioning

  • Captioning models are adapted to two-person SMPL-X inputs and evaluated with retrieval, multimodal-distance, and linguistic metrics.The evaluation follows the same train/test/validation split protocol as text-conditioned generation.Source: §A.1 · p. 13
  • TM2T achieves state-of-the-art performance across all reported human interaction captioning metrics.RAEs performs poorly on R Precision and linguistic metrics; the authors link this to difficulty modeling long-term dependencies between interactions and text.Source: §A.1 · p. 13

A.2. Causal order inference

The appendix reports results for causal-order inference, stylized interaction generation, and personality assessment, then describes SMPL-X fitting and Inter-X’s action categories.

A.3. Stylized human interaction generation

  • Actformer leads FID and accuracy, while MDM leads diversity and multi-modality in stylized interaction generation.Source: §A.3 · p. 13

A.4. Personality assessment

  • MS-G3D performs best on every personality element except Agreeableness, where CTR-GCN has the best R2 score.Source: §A.4 · p. 13

B. SMPL-X optimization details

A.2. Causal order inference

  • 76.8% is MS-G3D’s causal-order inference performance, the best among tested methods but still described as far from satisfactory.The reported task is binary classification of actor-versus-reactor order.Source: §A.2 · p. 13, Table A.2 · p. 14

C. The action categories

D. Samples of textual annotations

The section points to sample textual annotations from Inter-X in Figure E.1.

D. Samples of textual annotations

E. More visualization results

The section presents rendered RGB frames and additional visualizations of Inter-X.

E. More visualization results

Related papers

Loading 2312.16051v1…