PaperlayerView exact revision on arXiv

Source-linked AI summary

POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World

Boshen Xu, Sipeng Zheng, Qin Jin

arXiv:2403.05856v1cs.CV

TL;DR

Egocentric hand-object interaction recognition is difficult to adapt from third-person video, while egocentric data collection can be costly. POV learns view-agnostic representations using prompts and evaluates third-person-to-egocentric transfer on two benchmarks, where it outperforms other approaches.

  • Problem

    Current methods struggle to adapt hand-object interaction recognition from third-person to egocentric views, and geometric RGB approaches can require abundant views and struggle with fine-grained interactions.

    Source: §1 · pp. 1–2

  • Method

    POV pretrains on third-person video with interactive masking prompts and view-aware prompt tuning, then can optionally fine-tune on limited egocentric video.

    Source: §1 · p. 2, §3 · p. 3, §4.1 · p. 4

  • Results

    POV outperforms other approaches on all Ego-HOI-XView benchmarks, which evaluate cross-view egocentric interaction recognition using Assembly101 and H2O.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The two Ego-HOI-XView benchmarks provide settings for evaluating transfer from multiple third-person views to egocentric interaction recognition.

    Source: §1 · p. 2

    Geometric RGB approaches require sufficiently abundant views for satisfactory results and struggle to recognize fine-grained interactions.

    Source: §1 · p. 2

Abstract

from arXiv · show

We humans are good at translating third-person observations of hand-object interactions (HOI) into an egocentric view. However, current methods struggle to replicate this ability of view adaptation from third-person to first-person. Although some approaches attempt to learn view-agnostic representation from large-scale video datasets, they ignore the relationships among multiple third-person views. To this end, we propose a Prompt-Oriented View-agnostic learning (POV) framework in this paper, which enables this view adaptation with few egocentric videos. Specifically, We introduce interactive masking prompts at the frame level to capture fine-grained action information, and view-aware prompts at the token level to learn view-agnostic representation. To verify our method, we establish two benchmarks for transferring from multiple third-person views to the egocentric view. Our extensive experiments on these benchmarks demonstrate the efficiency and effectiveness of our POV framework and prompt tuning techniques in terms of view adaptation and view generalization. Our code is available at \url{https://github.com/xuboshen/pov_acmmm2023}.

1 INTRODUCTION

POV addresses cross-view egocentric hand-object interaction recognition by learning from multi-view third-person video with lightweight prompts. The paper introduces two Ego-HOI-XView benchmarks and reports that POV outperforms other approaches on them.

1 INTRODUCTION

  • Some existing Ego-HOI work requires bulky headset cameras, while geometric RGB methods require abundant views and struggle with fine-grained interactions.Ego-HOI has applications in egocentric robotics vision and virtual reality.Source: §1 · pp. 1–2
  • The motivating question is whether extensive multi-view third-person video can help models learn view-agnostic representations for egocentric hand-object interaction.Third-person instructional videos are described as rich and more readily available than egocentric recordings.Source: §1 · p. 1
  • POV learns view-agnostic representations from multi-view third-person videos using frame-level interactive masking prompts and lightweight token-level view-aware prompts.The framework includes third-person action understanding and view-agnostic prompt tuning, with optional egocentric fine-tuning.Source: §1 · p. 2
  • POV outperforms other approaches on all Ego-HOI-XView benchmarks, which use Assembly101 and H2O for cross-view egocentric interaction recognition.The benchmarks evaluate transfer from third-person videos to egocentric videos.Source: §1 · p. 2
  • The paper introduces frame- and token-level visual prompts for interaction focus and cross-view generalization, and constructs benchmarks on Assembly101 and H2O.The authors characterize POV as demonstrating generalization across different downstream Ego-HOI tasks.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses egocentric interaction recognition, third-person-to-egocentric transfer, view-agnostic representations, and visual prompting. POV focuses prompt tuning on learning view-agnostic representations for Ego-HOI from unsynchronized multi-view video.

2 RELATED WORK

  • Egocentric recognition models use 3D CNNs, recurrent networks, and multi-stream architectures, while third-person video has also been used for pretraining and domain adaptation.Prior approaches include Ego-Exo, SOS, and an unsupervised domain adaptation method for Ego-HOI.Source: §2 · p. 2
  • View-agnostic learning has used synchronized or unsynchronized multi-view data, geometric transformations, latent unseen views, and cross-view prediction.The paper distinguishes its focus on relationships among unsynchronized multi-view videos for Ego-HOI from earlier work on third-person body activities.Source: §2 · p. 2
  • Visual prompt tuning treats prompts as trainable continuous vectors; prior vision methods use prompt tokens to adapt pretrained models for image classification and other tasks.The cited examples also apply prompts to depth estimation, out-of-domain data, and tracking.Source: §2 · pp. 2–3
  • POV applies prompt tuning to view-agnostic representation learning for Ego-HOI, with interactive masking prompts and view-aware prompts in a two-stage third-person training framework.An optional third stage fine-tunes on limited egocentric video.Source: §2 · p. 3, Figure 2 · p. 3

3 METHOD OVERVIEW

Ego-HOI-XView evaluates transfer from third-person camera views to egocentric video, with optional egocentric tuning. POV uses lightweight visual prompts to adapt across views while limiting full-parameter tuning.

3 METHOD OVERVIEW

  • Ego-HOI-XView trains on third-person videos from multiple camera angles, optionally fine-tunes on zero-shot or few-shot egocentric data, and predicts in the target view.The task is framed as learning from others’ instructions before exploring egocentric scenarios.Source: §3 · p. 3
  • POV targets view-agnostic knowledge so the model can recognize hand-object interactions from different camera angles, including egocentric ones.The framework is proposed for the Ego-HOI-XView task.Source: §3 · p. 3
  • Because view-agnostic tuning has noisy objectives under limited data scale and diversity, POV uses visual prompt tuning to adapt across angles with fewer trainable parameters.The method includes two essential third-person learning tasks and an optional egocentric fine-tuning task.Source: §3 · p. 3

4 POV FRAMEWORK

POV pre-trains on third-person videos with action-understanding and view-agnostic prompt-tuning tasks, then optionally fine-tunes for egocentric scenarios. It uses frame-level interactive masks and token-level view-aware prompts to learn fine-grained, view-agnostic hand-object interaction representations.

4 POV FRAMEWORK

  • POV pre-trains on third-person videos with prompt-based action understanding and view-agnostic prompt tuning, then optionally fine-tunes for egocentric scenarios.The framework uses a vision transformer.Source: §4 · p. 3

4.1 Prompt-based Action Understanding

  • Frame-level interactive masks focus on hands and the interacting object; hard prompts use fixed Gaussian values, while soft prompts learn pixel values and are omitted at inference.An off-the-shelf detector localizes the hands and object during pre-training.Source: §4.1 · p. 4

4.2 View-agnostic Prompt Tuning

  • Trainable view-aware prompts are concatenated with video tokens across transformer blocks, while cross-view alignment minimizes embedding distance between an original view and an unseen view simulated by prompts.The authors report that visual prompt tuning is less prone to overfitting and learns view-agnostic representations.Source: §4.2 · p. 4

4.3 Egocentric Fine-Tuning

  • For novel views, POV combines view-aware prompts from third-person views into joint prompts; egocentric adaptation uses prompt tuning with unlabeled data or full tuning with limited labeled data.Joint prompts are used with novel-view video tokens in the transformer.Source: §4.3 · pp. 4–5

4.4 Training Strategy

  • Training first optimizes all model parameters on labeled third-person videos, then freezes pre-trained parameters and updates prompts; the view-alignment loss weight is set to 0.001.Egocentric fine-tuning is optional.Source: §4.4 · p. 5

5 EXPERIMENTS

Across zero-shot and few-shot cross-view benchmarks, POV outperforms the compared methods, and ablations support its third-person pretraining and prompt designs. The experiments also examine dataset setups and computational cost.

5.1 Experiment Setup

  • Assembly101 experiments use 134,104 third-person pretraining snippets, 10,388 egocentric snippets each for fine-tuning and testing, and 20,766 unseen-view snippets; H2O uses 2,764 third-person, 180 fine-tuning, and 511 test snippets.The benchmarks evaluate verb, object, and action accuracy under zero-shot or few-shot adaptation.Source: §5.1 · p. 5

5.2 Comparison with State of the Arts

  • POV improves top-1 action accuracy by at least +2.65% on Assembly101 and +3.51% on H2O, and improves top-1 verb and action accuracy by +50.91% and +45.53% on Assembly101 over EgoVLP.The reported comparisons cover zero-shot and few-shot Ego-HOI-XView settings; the paper also summarizes advantages under both setups.Source: §5.2 · p. 6

5.3 Ablation Study

  • POV exceeds 3DTRL on verb, noun, and action accuracy for unseen third-person views, while supervised egocentric fine-tuning improves top-1 action accuracy by +39.28% on Assembly101 and +29.69% on H2O.Self-supervised egocentric fine-tuning gives smaller top-5 accuracy improvements of +4.17% and +3.12%, respectively.Source: §5.3 · pp. 6–7
  • Cross-view alignment and interactive masking prompts together improve top-5 accuracy over baseline by +27.67% for verbs, +12% for nouns, and +4.93% for actions, although object accuracy drops when both are used.The authors suggest a trade-off between cross-view generalization and view-specific discrimination.Source: Table 5 · p. 6, §5.3 · p. 7
  • Interactive masking prompts perform best with either an appropriate mask size and hard prompts or a small patch size and soft prompts.The authors attribute the soft-prompt result to adaptable emphasis on the in-contact hand and object, and the hard-prompt result to better localization of the HOI region.Source: §5.3 · p. 7, Table 6 · p. 7
  • Block-level view-aware prompts outperform model-level and layer-level prompts in top-5 accuracy, while layer-level prompts tend to overfit uninformative noise.The authors characterize block-level prompts as a shallow, high-level representation design.Source: §5.3 · p. 7, Table 7 · p. 7

5.4 Computation Efficiency Analysis

  • POV uses approximately 20% less GPU memory than full tuning while balancing generalization and computational cost.The comparison reports learnable parameters, GPU memory, and top-5 accuracy for different tuning methods.Source: §5.4 · p. 8, Table 8 · p. 8

6 QUALITATIVE ANALYSIS

The qualitative analysis reports classwise gains and more compact within-class, separated between-class embeddings for POV than for MViT-3rd. Examples illustrate improved fine-grained predictions in H2O and monochrome Assembly101 scenes.

Per Class Analysis

  • POV improves per-class accuracy over MViT-3rd in most HOI categories, with exceptions including “grab” and “apply” and their noun combinations.The authors describe these classes as ambiguous and difficult to distinguish.Source: §6 · p. 8

Feature Cluster Analysis

  • Compared with MViT-3rd, POV embeddings are more compact within classes and more scattered between classes.The analysis covers verb, noun, and action classes; it also reports more centralized spatial than motion representations for certain classes.Source: §6 · p. 8

Prediction Examples

  • In examples from H2O and Assembly101, POV correctly recognizes the fine-grained action class place lotion, misclassified as close lotion by MViT-3rd, and improves recognition in monochrome Assembly101 environments.Source: §6 · p. 8

7 CONCLUSION

POV pre-trains on third-person videos for action understanding and view-agnostic tuning, then can be fine-tuned on egocentric videos. The authors report that it outperforms existing approaches across evaluation setups and suggest expanding pre-training with view labels.

7 CONCLUSION

  • POV pre-trains on third-person videos for action understanding and view-agnostic tuning, with optional egocentric fine-tuning for view adaptation.The framework uses a prompt-oriented vision transformer with frame- and token-level visual prompts.Source: §7 · p. 8
  • Across different evaluation setups, the authors report POV is effective and efficient and outperforms existing approaches.Source: §7 · p. 8
  • The authors suggest expanding pre-training data with view labels, obtainable using an off-the-shelf view classifier, as a future direction for better view-agnostic representation.Source: §7 · p. 8

A ADDITIONAL DATASET DETAILS

The dataset details describe the camera-view splits, selected action categories, and egocentric fine-tuning subsets used for Assembly101 and H2O.

A ADDITIONAL DATASET DETAILS

  • Both datasets use four third-person views for pre-training, then egocentric video for fine-tuning and Ego-HOI-XView evaluation; Assembly101 adds two views for HOI-XView evaluation.The camera views are synchronized and fixed, but synchronization information is not used.Source: A ADDITIONAL DATASET DETAILS · p. 11
  • Assembly101’s 1,380 classes are long-tailed, so the study selects 142 fine-grained head categories formed from 20 verbs and 24 nouns.Source: A ADDITIONAL DATASET DETAILS · p. 11
  • H2O has a uniform category distribution, with fine-grained actions formed by combining 11 verbs and 8 nouns.Source: A ADDITIONAL DATASET DETAILS · p. 11
  • Egocentric fine-tuning subsets average 73 samples per action class for Assembly101 and 5 for H2O.Source: A ADDITIONAL DATASET DETAILS · p. 11

B ADDITIONAL EXPERIMENTAL DETAILS

The appendix provides implementation details for zero-shot Ego-HOI-XView recognition, one of three additional experiment setups. It describes domain-adaptation baselines, optimization settings, and Assembly101 category labels.

B.1 Zero-shot Ego-HOI-XView Recognition

  • Zero-shot Ego-HOI-XView comparisons include methods fine-tuning most parameters and Tent, which fine-tunes only a small number.SHOT fine-tunes the model except its classification head; AaD fine-tunes all parameters, and Tent adjusts Layer Normalization transformation parameters.Source: §B.1 · p. 11
  • The described domain-adaptation methods use AdamW with an initial learning rate of 1e-4 and cosine scheduling to a final 1e-6.Source: §B.1 · p. 11

B ADDITIONAL EXPERIMENTAL DETAILS

B.2 Few-shot Ego-HOI-XView Recognition

The appendix details few-shot Ego-HOI-XView recognition and 3rd-to-ego HOI recognition, including the training setups and the datasets’ view splits. Figures 6 and 7 illustrate the POV training and inference pipelines for Assembly101 and H2O.

B.2 Few-shot Ego-HOI-XView Recognition

  • Few-shot Ego-HOI-XView recognition fine-tunes models on labeled egocentric videos using full parameters, comparing POV with EgoVLP and Ego-Exo.EgoVLP is pre-trained on over 3.8 million Ego4D video-text pairs; Ego-Exo is pre-trained on over 1 million Kinetics-400 third-person videos.Source: §B.2 · p. 11
  • Figures 6 and 7 illustrate POV’s training and inference pipelines and view splits for Assembly101 and H2O, respectively.Source: Figure 6 · p. 12, Figure 7 · p. 12
  • Table 10 lists H2O verb and noun categories; the few-shot setup uses a pre-trained SlowFast ResNet50 for further fine-tuning on limited egocentric videos.It is fine-tuned for 30 epochs with SGD, starting at 1e-2 and dropping the learning rate by 0.1 at epochs 20 and 25.Source: Table 10 · p. 12, §B.2 · p. 12

B.3 3rd-to-ego HOI Recognition

  • The 3rd-to-ego task predicts hand-object interactions directly in egocentric videos without egocentric fine-tuning after third-person pre-training.Its comparisons include 3DTRL, an explicit geometric transformation layer, and EgoVLP, which classifies by matching video-text similarities to action-class text.Source: §B.3 · p. 12

C QUANLITATIVE RESULTS

The qualitative results report improved verb and noun recognition, while noting remaining challenges with temporally dependent verb distinctions and some Assembly101 domains.

C QUANLITATIVE RESULTS

  • POV enhances verb and noun recognition, but distinguishing verbs such as “grab” and “place” remains challenging because predictions strongly rely on temporal information.Hand-object interaction recognition also remains difficult in specific Assembly101 domains.Source: C QUANLITATIVE RESULTS · p. 12
  • Figures 8 and 9 present Assembly101 distributions of verbs, nouns, and actions.Source: Figure 8 · p. 13, Figure 9 · p. 13
  • Figures 10 and 11 show examples of egocentric predictions on H2O and Assembly101.Source: Figure 10 · p. 14, Figure 11 · p. 14
Loading 2403.05856v1…