Learning from Semantic Alignment between Unpaired Multiviews for Egocentric Video Recognition

• Very few first-person videos have paired third-person counterparts, limiting the scale and scope of multiview learning.Source: §1 · p. 1
• Most first-person videos share only partial semantics with available unpaired third-person views, such as the same verb or object but not both.Source: Figure 1 · p. 1, §1 · p. 2
• SUM-L builds high-similarity multiview pseudo-pairs for semantics-aware view-invariant alignment and also uses video-text alignment for first- and third-person videos.Source: §1 · p. 2
• The paper frames this as a new problem: using unpaired third-person videos to help first-person learning when cross-view semantics are only partially similar.Source: §1 · pp. 1–2
• SUM-L is competitive on Charades-Ego, EPIC-Kitchens, and EPIC-Kitchens-100 against contrastive- and triplet-loss view-alignment methods.Source: §1 · p. 2

2.1. Egocentric Video Learning

• Prior egocentric-learning work uses first-person data, off-the-shelf model supervision, or large-scale pretraining; Li et al. [22] also rely heavily on off-the-shelf models.Source: §2.1 · pp. 2–3

2.2. First-person Video Representation Learning with Joint Third-person Videos

• Paired-view studies [39] [40] show potential for joint representations, but paired videos remain scarce and existing alignment methods are described as less effective with unpaired third-person data.Source: §2.2 · p. 3

2.3. Unpaired Multimodal Learning

• Kundu et al. align unpaired video and pose data assuming identical semantics, whereas this paper studies alignment when cross-view semantics are only partly similar.Source: §2.3 · p. 3

3.1. Semantics-based Pseudo-pair Mining

• SUM-L mines the most textually similar third-person video for each first-person video to construct cross-view pseudo-pairs.Textual narrations are encoded as vectors, and cosine similarity is used to compare them.Source: §3.1 · p. 3

3.2. Typical Multiview Alignment

• Typical contrastive alignment minimizes distances for paired features, but identical semantics are scarce in this setting and naive alignment may yield suboptimal first-person representations.Contrastive optimization can also require large batches and many training epochs.Source: §3.2 · pp. 3–4

3.3. Semantics-based Multiview Learning

• SUM-L aligns cross-view pseudo-pairs with relatively high textual semantic similarity, weighting pairs with higher similarity more heavily.This makes the cross-view alignment semantics-aware rather than treating pairs with different semantic similarities equally.Source: §3.3 · p. 5
• Video-text contrastive alignment lets all first- and third-person videos contribute learning signals, including first-person samples in low-similarity pseudo-pairs.The method aligns video features with their corresponding textual description vectors.Source: §3.3 · p. 5

3.4. Training and Evaluation Pipelines

• Training combines first- and third-person classification losses with cross-view and cross-modal losses, after initializing and pretraining the third-person encoder and task head.Evaluation uses the first-person encoder and task-specific head for first-person video predictions.Source: §3.4 · p. 5

4.1.1 Dataset

• Experiments use LEMMA third-person videos as unpaired samples, Charades-Ego first-person videos, and EPIC-Kitchens and EPIC-Kitchens-100 to validate the approach.LEMMA includes single- and multi-agent videos, but only single-agent third-person videos are used as unpaired samples; Charades-Ego uses first-person videos for training and inference.Source: §4.1.1 · pp. 5–6

4.1.2 Baseline Methods

• Baselines include first-person-only training, typical contrastive learning, and triplet learning, with the latter two using the framework’s pseudo-pair construction and inference strategy.Typical contrastive learning and triplet learning use the same implementation protocol as SUM-L where specified.Source: §4.1.2 · p. 6

4.1.3 Implementation Details

• All baselines and SUM-L use identical hyperparameters; dataset settings vary in learning rate, batch size, and epoch limit.Charades-Ego uses a 0.25 base learning rate, batch size 32, and 60 epochs; EPIC-Kitchens datasets use a 0.01 base learning rate, weight decay 0.0001, batch size 64, and 30 epochs.Source: §4.1.3 · p. 6

• +1.6% mAP on Charades-Ego and +1.2%/+1.5% EPIC-Kitchens verbs/nouns top-1 accuracies over FPV Only with R3D-50.Source: Table 1 · p. 6, §4.2 · p. 6
• SlowFast results reconfirm effective alignment with partially shared semantics and more significant, consistent improvement over existing multiview alignment methods.The evaluation covers Charades-Ego, EPIC-Kitchens, and EPIC-Kitchens-100.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 7
• +1.2% mAP over FPV only with [22] pretrained weights establishes a new Charades-Ego state-of-the-art.Source: Table 2 · p. 7, §4.2 · p. 7
• +1.9%/+1.1% verbs/nouns top-1 accuracies over FPV Only on EPIC-Kitchens, and +2.9%/+1.8% over existing alignment methods.Source: Table 3 · p. 7, §4.2 · p. 7
• On EPIC-Kitchens-100, SUM-L ranks first in noun accuracy and third in verb accuracy among CNN-backbone state-of-the-art methods.Source: Table 4 · p. 7, §4.2 · p. 7

4.3.1 Impact of Different Alignment Tasks

• Cross-view alignment adds 0.9% mAP over FPV Only; video-text alignment adds 0.7% mAP, while removing semantic weighting costs 0.2%.These task-combination ablations use the Charades-Ego dataset and ResNet-3D-50.Source: §4.3.1 · p. 7, §4.3 · p. 7
• Figure 4 compares FPV Only and SUM-L activation maps for cross-view similar semantics labeled “holding pan” and “take.”Source: Figure 4 · p. 8

4.3.2 Impact of Third-person View Backbone

• Freezing the third-person encoder drops mAP by 0.5%, shared encoder weights by 1.0%, and identical Kinetics-400 initialization by 0.2%.The original setup outperforms each alternative implementation.Source: §4.3.2 · p. 8, Table 6 · p. 8

• Compared with FPV Only, SUM-L responds more to positive spatio-temporal locations in first-person clips.On EPIC-Kitchens, the compared encoders have 61.1%/45.8% versus 62.3%/47.3% top-1 verb/noun accuracies.Source: §4.4 · p. 8
• SUM-L also responds strongly to third-person locations with semantics overlapping the first-person clip, indicating use of unpaired third-person knowledge.Source: §4.4 · p. 8

5. Conclusions

• SUM-L distills knowledge from unpaired, cross-dataset third-person video without requiring exact same multiview semantic information.The paper frames this as a new problem in first-person and third-person video learning.Source: §5 · p. 9
• Under a more challenging scenario than typical paired or unpaired multimodal or multiview learning, SUM-L outperforms existing view-alignment methods.Source: §5 · p. 9
• The improvements are not very significant; the authors suggest relatively little third-person data as one possible reason and identify more datasets and downstream tasks as future work.Unpaired cross-view retrieval is one example of a downstream task they propose to study.Source: §5 · p. 9