PaperlayerView exact revision on arXiv

Source-linked AI summary

Identity-Aware Human-Object Interaction Motion Captioning

Yiming Wang, Yonghao Dang, Huilai Li, Jiawei Tu, Jianqin Yin

arXiv:2608.20690v1cs.CVcs.AI

TL;DR

Existing HOI motion captions describe interactions without grounding them in subject identity, despite identity-dependent applications requiring the correct individual to be associated with the motion. The paper introduces an identity-aware HOI captioning task and ID-HOINet, which learns identity and motion features from multi-view videos and rewrites identity-agnostic captions with retrieved identities. Experiments report state-of-the-art performance, while the current evaluation remains limited to 18 subjects and single-subject, single-object interactions.

  • Problem

    Existing HOI motion captioning methods typically describe what interaction occurs without specifying who performs it, limiting explicit identity-motion association in identity-dependent scenarios.

  • Method

    ID-HOINet learns identity and interaction-motion features from multi-view videos, then uses TSCR to retrieve identity, generate an identity-agnostic caption, and rewrite it with the predicted identity.

  • Results

    ID-HOINet achieves state-of-the-art performance on the proposed identity-aware HOI motion captioning task.

  • Takeaways & Limitations

    The task and framework explicitly associate subject identities with corresponding human-object interaction motions in generated captions.

  • Takeaways & Limitations

    Experiments use reorganized data with 18 subjects and focus on single-subject, single-object interactions, leaving larger populations, diverse scenes, and more complex interactions for future work.

Abstract

from arXiv · show

Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates "Sub_ID lifts the chair" rather than "A person lifts the chair". For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance.

Introduction

Existing HOI motion captioning describes interactions without identifying the performer, limiting its use in identity-dependent settings. The paper introduces an identity-aware task and ID-HOINet, which separates identity retrieval from motion captioning before rewriting the result.

  • Existing HOI captioning methods describe the interaction but typically omit the identity of the subject performing it.
  • Identity-agnostic captions are inadequate for scenarios requiring motion to be associated with the correct individual, including personalized rehabilitation monitoring and multisubject motion analysis.
  • The proposed task requires captions to explicitly associate a subject identity with the corresponding HOI motion, such as “Sub_ID lifts the chair.”
  • The dataset reorganizes multi-view videos from BEHAVE and InterCap into temporally complete HOI segments with identity-aware annotations and subject-disjoint training and test identities.
  • ID-HOINet combines MVIML for learning identity and interaction-motion features with TSCR, which retrieves identity, generates an identity-agnostic caption, and rewrites it into the final caption.
  • ID-HOINet achieves state-of-the-art performance on the proposed identity-aware HOI motion captioning task.

Related Work

HOI motion captioning methods model interactions, motion, and video-language information but generally identify what happens rather than who performs it. Existing identity-aware captioning methods address films and sports, yet do not explicitly connect identities to complex human-object interactions involving varying objects.

  • HOI motion captioning research models object interactions, human motion, and general video understanding to generate natural-language descriptions.
  • Existing methods generally produce identity-agnostic captions that specify the performed HOI motion but not the performer.
  • Identity-aware video captioning extends descriptions by associating observed events with subject identities, primarily in films and sports events.
  • Those methods are not specifically designed for temporally complete human-object interactions involving diverse object categories and joint identity-motion reasoning.

Dataset Processing

The paper restructures BEHAVE and InterCap into temporally complete, identity-aware HOI segments and partitions them for gallery-based evaluation. Training uses synchronized multi-view data, while testing unfolds query views into independent single-view instances.

  • BEHAVE and InterCap provide synchronized multi-view human-object interactions and identities but lack identity-aware HOI motion captions, requiring dataset restructuring.
  • Each synchronized multi-view video is segmented by annotated start and end timestamps into temporally complete interaction units with identity-agnostic motion captions.
  • Subjects are mapped to 18 global labels, Sub01 through Sub18, that remain consistent across motion segments and camera views.
  • The processed dataset contains 2,073 multi-view HOI segments, with 1,700 training segments from 10 subjects and 373 testing segments from 8 subjects.
  • Testing selects one gallery segment per test subject, leaving 365 query segments whose four camera views yield 1,460 independent single-view query instances.

Method

ID-HOINet learns identity and interaction-motion features from synchronized multi-view videos, then uses a two-stage testing strategy to generate identity-aware captions from single-view queries.

  • Multi-View Identity-Motion Learning: The Video Encoder extracts sampled-frame features, which MVIML processes to learn identity and interaction-motion representations.MVIML uses shared temporal-view features and separate identity and motion branches.
  • Multi-View Identity-Motion Learning: MVIML projects multi-view features into temporal-view tokens, where each token represents an interaction at a temporal stage and camera viewpoint.The module incorporates temporal and viewpoint positions before modeling identity and motion information.
  • Multi-View Identity-Motion Learning: An identity token captures global subject characteristics across observations, while an identity-specific transformer produces the identity feature.The identity token is concatenated with temporal-view features, and the encoded identity-token output is selected.
  • Multi-View Identity-Motion Learning: A motion-query branch models relations between temporal-view features and motion tokens to produce an interaction-motion feature.The motion-specific transformer outputs a motion representation containing temporal-view and motion-query tokens.
  • Two-Stage Caption Rewriting: TSCR first matches a single-view query to a gallery identity and independently generates an identity-agnostic caption, then rewrites it with the retrieved identity.The second stage uses a pretrained Caption Rewriting Decoder to produce the final identity-aware caption.
  • Two-Stage Caption Rewriting: During training, the Identity Decoder and Identity-Agnostic Caption Decoder are trained in parallel to provide the outputs required by TSCR.The Identity Decoder maps identity features to subject entities, while the caption decoder maps motion features to identity-agnostic HOI captions.

Experiments

The experiments evaluate caption quality, identity recognition, baseline performance, ablations, and qualitative identity–motion association.

  • Experimental Design: The study compares ID-HOINet with representative video captioning methods, conducts component ablations, and uses qualitative visualizations of identity–motion association.These experiments evaluate effectiveness and illustrate the model’s generated captions.
  • Evaluation Metrics: The evaluation combines caption-quality metrics with identity-recognition metrics to assess identity-aware HOI motion captioning.Caption quality is measured using BLEU-4, METEOR, ROUGE-L, and CIDEr; identity recognition uses ID Acc and Weighted ID Acc.
  • Caption Quality: BLEU-4, METEOR, ROUGE-L, and CIDEr measure n-gram precision, semantic alignment, sequence similarity, and caption consensus, respectively.BLEU-4 evaluates up to four-gram precision, METEOR unigram-level alignment, ROUGE-L longest-common-subsequence similarity, and CIDEr n-gram consensus.
  • Identity Recognition: ID Acc treats all query samples equally, whereas Weighted ID Acc averages per-identity accuracies to reduce imbalance effects.The weighted metric first computes accuracy for each identity before averaging across identities.

Baselines

ID-HOINet is compared with five representative captioning baselines under a shared evaluation setting, using a multi-view implementation designed for single-view inference.

  • Baselines: ID-HOINet is compared with CLIP-Captioner, CARE, NACF, CoCap, and SwinBERT under the same evaluation setting.The comparison includes five representative video captioning methods.
  • Implementation Details: The implementation samples 32 temporal steps from 4 camera views using pre-extracted 768-dimensional CLIP visual features.The model is implemented in PyTorch.
  • Implementation Details: MVIML uses one learnable identity token and eight learnable motion-query tokens processed by separate Transformer branches.The branches support identity recognition and identity-agnostic captioning, respectively.
  • Implementation Details: Training runs for 50 epochs with AdamW optimization, scheduled learning rates, batch size 32, and a single NVIDIA GeForce RTX 4090 GPU.The main network and trainable decoder use learning rates of 5×10^-5 and 1×10^-5, respectively, with five-epoch warmup.

Datasets

The study uses BEHAVE and InterCap multi-view HOI datasets, and reports consistent ID-HOINet gains across captioning and identity-matching metrics.

  • BEHAVE: BEHAVE contains 321 four-camera video sequences of eight subjects interacting with 20 everyday objects across five indoor environments.It records diverse full-body hand-object and foot-object interactions.
  • InterCap: InterCap contains 223 six-camera HOI videos featuring ten subjects and ten everyday objects with varied sizes and affordances.Its RGB videos include diverse full-body hand-object and foot-object interactions.
  • Results: ID-HOINet consistently outperforms representative video captioning methods across captioning and identity-matching metrics.The reported improvements indicate more accurate interaction captions while maintaining reliable subject identification.

Ablation Study

Ablations show that multi-view learning and randomized-view training support single-view testing, while direct caption generation trades identity reliability for textual similarity.

  • MVIML: Removing MVIML substantially degrades all metrics, especially ID Acc and Weighted ID Acc.The result underscores the importance of multi-view information for jointly understanding identity and HOI motion.
  • View Randomization: Removing view randomization reduces both captioning and identity-matching performance under single-view testing.The ablation indicates that randomized multi-view training improves robustness in the single-view setting.
  • Two-Stage Generation: Direct identity-aware generation raises captioning scores but lowers ID Acc from 72.94 to 67.39 and Weighted ID Acc from 58.31 to 52.75.The comparison reveals a trade-off between textual similarity and identity reliability.
  • Two-Stage Generation: The proposed two-stage strategy is adopted because assigning an accurate HOI motion caption to the wrong subject compromises semantic validity.The authors therefore prioritize identity reliability over direct-generation captioning scores.

Qualitative Results

ID-HOINet more consistently associates subject identity, interacted object, and HOI motion than representative video captioning methods in qualitative comparisons.

  • Qualitative Results: ID-HOINet correctly identifies Sub07 and describes the square table interaction as lifting and putting it down.The baselines variously confuse the object, hands, motion sequence, or subject.
  • Qualitative Results: ID-HOINet correctly associates Sub18 with sitting on the stool, while comparison methods misidentify the subject and often predict lifting.Despite adding motion details, ID-HOINet preserves the correct identity and core HOI semantics.
  • Qualitative Results: Figure 3 compares sampled frames, ground-truth captions, and generated captions from ID-HOINet and representative video captioning methods.Correct and incorrect identity, object, and motion expressions are highlighted in green and red, respectively.
  • Qualitative Results: The examples show better consistency among subject identity, interacted object, and HOI motion than video captioning baselines.They also indicate that successful predictions must associate the observed interaction with the corresponding subject, beyond lexical similarity.

Limitations

The study is limited by its small, relatively narrow dataset scope and its focus on single-subject, single-object interactions.

  • Limitations: The experiments use reorganized BEHAVE and InterCap data containing only 18 subjects and a limited range of objects, actions, and capture environments.Generalization to larger subject populations and more diverse real-world scenes remains to be investigated.
  • Limitations: The current setting excludes more complex multi-person or multi-object interactions.Future work proposes extending the task to these interaction settings.

Conclusion

The paper introduces identity-aware HOI motion captioning and ID-HOINet, which learns identity and motion features from multi-view videos and rewrites captions with inferred identities.

  • Conclusion: The proposed task requires captions to describe both subject identity and the corresponding human-object interaction motion.The task is designed using processed BEHAVE and InterCap data with temporal segmentation, identity-aware annotation, and data partitioning.
  • Conclusion: ID-HOINet combines multi-view identity-motion learning with a two-stage caption rewriting strategy.The strategy determines query identity through gallery matching, generates an identity-agnostic motion caption, and rewrites it with the determined identity.
  • Conclusion: Experimental results demonstrate ID-HOINet’s effectiveness in associating subject identities with corresponding HOI motions in generated captions.Future work targets multi-person and multi-object interactions and finer-grained motion descriptions.
Loading 2608.20690v1…