PaperlayerView exact revision on arXiv

Source-linked AI summary

Synchronization is All You Need: Exocentric-to-Egocentric Transfer for Temporal Action Segmentation with Unlabeled Synchronized Video Pairs

Camillo Quattrocchi, Antonino Furnari, Daniele Di Mauro, Mario Valerio Giuffrida, Giovanni Maria Farinella

arXiv:2312.02638v3cs.CV

TL;DR

Exocentric temporal action segmentation systems can struggle on wearable-camera videos, while supervised adaptation requires costly egocentric collection and labeling. The paper uses knowledge distillation with unlabeled synchronized exocentric-egocentric pairs, reporting improvements on Assembly101 and EgoExo4D without egocentric labels.

  • Problem

    Exocentric TAS systems can retain a domain gap on egocentric video, while supervised adaptation requires costly, time-intensive collection and labeling of egocentric videos.

    Source: §1 · p. 2

  • Method

    The method adapts an exocentric TAS model through knowledge distillation at the feature-extractor and downstream TAS-model levels using unlabeled synchronized exocentric-egocentric video pairs.

    Source: §3 · p. 6, §3.1 · p. 6

  • Results

    +15.99 edit score on Assembly101 and +3.32 on EgoExo4D are reported over an exocentric-only baseline, with performance on par with supervised approaches trained on labeled egocentric data.

    Source: §6 · p. 14

  • Takeaways & Limitations

    The reported results indicate that synchronized unlabeled pairs can support exocentric-to-egocentric TAS adaptation without egocentric labels.

    Source: §1 · p. 3

Abstract

from arXiv · show

We consider the problem of transferring a temporal action segmentation system initially designed for exocentric (fixed) cameras to an egocentric scenario, where wearable cameras capture video data. The conventional supervised approach requires the collection and labeling of a new set of egocentric videos to adapt the model, which is costly and time-consuming. Instead, we propose a novel methodology which performs the adaptation leveraging existing labeled exocentric videos and a new set of unlabeled, synchronized exocentric-egocentric video pairs, for which temporal action segmentation annotations do not need to be collected. We implement the proposed methodology with an approach based on knowledge distillation, which we investigate both at the feature and Temporal Action Segmentation model level. Experiments on Assembly101 and EgoExo4D demonstrate the effectiveness of the proposed method against classic unsupervised domain adaptation and temporal alignment approaches. Without bells and whistles, our best model performs on par with supervised approaches trained on labeled egocentric data, without ever seeing a single egocentric label, achieving a +15.99 improvement in the edit score (28.59 vs 12.60) on the Assembly101 dataset compared to a baseline model trained solely on exocentric data. In similar settings, our method also improves edit score by +3.32 on the challenging EgoExo4D benchmark. Code is available here: https://github.com/fpv-iplab/synchronization-is-all-you-need.

1 Introduction

Temporal action segmentation divides videos into action intervals, but exocentric systems often perform poorly on wearable-camera footage because of the domain gap. The paper proposes adapting them with unlabeled synchronized exocentric-egocentric video pairs and reports improvements across Assembly101 and EgoExo4D.

Introduction

  • Temporal action segmentation divides a video into meaningful intervals corresponding to distinct actions or activities, with applications including robotics and activities of daily living.Egocentric video can provide a privileged view of human-object interactions and object manipulation.Source: §1 · pp. 1–2
  • Exocentric TAS systems often perform poorly on wearable-camera videos because the viewpoints create a domain gap, while supervised adaptation requires costly, time-intensive egocentric collection and labeling.Unsupervised domain adaptation uses labeled exocentric and unlabeled egocentric videos, but can retain residual domain shift.Source: §1 · p. 2
  • +15.99 and +11.64 edit score on Assembly101 and up to +3.32 edit score on EgoExo4D are reported for the proposed method over exocentric-only baselines.The paper also reports performance on par with supervised approaches trained on labeled egocentric data, without using egocentric labels.Source: §1 · p. 3
  • The proposed adaptation uses unlabeled synchronized exocentric-egocentric video pairs and knowledge distillation at both the feature-extractor and downstream TAS-model levels.The pairs need no temporal action segmentation annotations; the paper describes them as inexpensive to collect in application settings such as factories with fixed cameras.Source: §1 · pp. 2–3

2 Related Work

Prior work covers temporal action segmentation, domain adaptation, exocentric-to-egocentric transfer, and knowledge distillation, but the paper identifies synchronized unlabeled video pairs for exocentric-to-egocentric TAS adaptation as uninvestigated. It applies distillation at both feature and TAS-model levels.

Temporal Action Segmentation

  • Temporal action segmentation approaches include temporal-convolutional encoder-decoders, multistage refinement, recurrent models, and C2F-TCN; the paper follows prior work in using C2F-TCN.The cited literature also includes temporal-resolution ensembles, higher-level representations, temporal-logic constraints, and diffusion-based refinement.Source: §2 · pp. 3–4

Exocentric-to-Egocentric Model Transfer

  • The paper identifies its work as the first systematic investigation of unlabeled synchronized exocentric-egocentric pairs for exocentric-to-egocentric transfer in temporal action segmentation.Earlier exocentric-to-egocentric transfer work addressed other tasks or lacked synchronized pairs; prior TAS domain-adaptation work considered environment shifts and unpaired inputs.Source: §2 · pp. 4–5

Domain Adaptation

  • Unsupervised domain adaptation trains with labeled source and unlabeled target data, but prior video work largely addressed action recognition across environments or subjects rather than synchronized exocentric-to-egocentric TAS.The paper states synchronized pairs provide a strong self-supervision signal; it also reports that its method works when videos are not perfectly synchronized.Source: §2 · pp. 3–5

Knowledge Distillation

  • Knowledge distillation has used logits or intermediate activations to guide a student, while this work applies it to exocentric-to-egocentric transfer using synchronized unlabeled pairs.The paper investigates distillation at both feature-extraction and TAS-model levels.Source: §2 · p. 5

3 Methodology

The task adapts exocentric temporal action segmentation using labeled exocentric videos and unlabeled synchronized exocentric–egocentric pairs. The proposed approach distills knowledge from an exocentric teacher to an egocentric student at both feature-extractor and TAS-model levels.

3 Methodology

  • The adaptation task trains a TAS algorithm using labeled exocentric videos and synchronized, unlabeled exocentric–egocentric pairs, then generalizes to egocentric test videos.The test segmentations are used only for evaluation.Source: §3 · p. 6

3.1 Proposed Adaptation Approach

  • A teacher trained on labeled exocentric data is fixed while a same-architecture student learns from synchronized pairs without exocentric or egocentric labels.The teacher provides the only supervision during distillation.Source: §3.1 · p. 6
  • Feature distillation minimizes mean squared error between teacher and student features extracted from paired exocentric and egocentric video segments.The student is encouraged to produce egocentric representations similar to the teacher’s exocentric representations.Source: §3.1 · p. 7
  • TAS-model distillation minimizes mean squared error between student and teacher internal representations across model layers on paired videos.The student TAS module is matched to representations produced by the teacher on the paired exocentric video.Source: §3.1 · p. 7

4 Experimental Settings

Experiments use synchronized multi-view datasets to evaluate adaptation from exocentric to egocentric video. Assembly101 supplies standardized assembly views, while EgoExo4D adds procedural cooking videos with less standardized camera placement.

Assembly101 Dataset

  • Assembly101 experiments use exocentric views v3 or v4 to adapt to egocentric view e4, with labeled exocentric training videos and separate adaptation and test pairs.The selected split includes 143 exocentric training videos, 89 adaptation pairs, and 44 paired test videos; 90 egocentric videos are selected for training.Source: §4 · pp. 8–9

EgoExo4D Dataset

  • EgoExo4D experiments select five cooking scenarios and 233 videos, using egocentric video and the human-annotated best exocentric view.The dataset is described as more challenging than Assembly101 because camera placement varies between videos.Source: §4 · pp. 8–9

Evaluation and implementation

  • The experiments use C2F-TCN with TSM or DINOv2 features and report edit score, F1 score, and Mean over Frames.Assembly101 uses coarse action labels and excludes the background class during training and evaluation.Source: §4 · pp. 8–9

5 Results and Discussion

Across view-transfer settings, distillation improves over no adaptation and can approach or exceed egocentric oracle results. On EgoExo4D, it also improves over unsupervised domain-adaptation and video-alignment approaches.

5.1 Performance of the Proposed Approach

  • +15.99 edit score over the no-adaptation baseline, while the best DINOv2 adaptation closely matches the best ego-oracle (28.59 vs 29.25 edit score).It also outperforms the ego-oracle using the same DINOv2 features (28.59 vs 26.42 edit score).Source: Table 1 · p. 10, §5.1 · p. 11
  • +24.88 MoF, +20.32 F110, and +14.86 edit score are achieved by the best TSM configuration with both distillation modules enabled.TAS-model distillation alone raises edit score to 20.17 from 10.53 and 12.38 without it.Source: Table 1 · p. 10, §5.1 · p. 11
  • With the less informative v4 source view, combined DINOv2 feature and TAS distillation improves over baselines by +18.74 MoF, +14.39 F125, and +11.64 edit score.TAS distillation alone brings edit score to 24.23, closer to the ego-oracle's 26.42.Source: Table 2 · p. 10, §5.1 · p. 11

5.2 Comparisons with the State of the Art

  • The proposed distillation approach improves over second-best results by +10.75 in edit score and +13.64 in MoF, highlighting unlabeled synchronized video pairs as supervision.DCORAL obtains an edit score of 17.13 versus 12.60 (+4.53) for No Adaptation.Source: §5.2 · pp. 12–13

5.3 Amounts of Video Pairs and Accuracy of Synchronization

  • About 5 hours of synchronized pairs—50% of the data—are enough for the method to double baseline results.It remains advantageous over no adaptation despite imperfect synchronization, including a 3% frame-drop rate (about 1 frame/sec at 30 fps).Source: Table 4 · p. 13, §5.3 · p. 13

5.4 Analysis on the EgoExo4D Dataset

  • +3.32 Edit, +5.91 MoF, and positive increments on other measures are reported on EgoExo4D; distillation reaches 23.88 edit versus 21.64 for the EXO oracle.The EGO oracle has 28.74 Edit, above the EXO oracle's 21.64, in this setting.Source: §5.4 · pp. 13–14, Table 6 · p. 14

5.5 Qualitative Results

  • Figure 4 presents qualitative temporal segmentations from different approaches on a test video; the compared methods use DINOv2 features.The paper points to supplementary material for more examples and discussion.Source: Figure 4 · p. 14, §5.5 · p. 14

5.4 Analysis on the EgoExo4D Dataset

  • Ego-teacher distillation also improves exocentric adaptation to 20.49 edit score, +4.44 over the no-adaptation baseline.The result is described as comparable to the EXO oracle's 21.64 edit score.Source: §5.4 · pp. 13–14

6 Conclusion

The paper concludes that exocentric temporal action segmentation can transfer to egocentric video using labeled exocentric data and unlabeled synchronized video pairs. Its appendices detail feature- and TAS-model distillation and the Assembly101 views used in experiments.

6 Conclusion

  • +15.99 edit-score improvement on Assembly101 and +3.32 on EgoExo4D over an exocentric-only baseline; the approach performs on par with supervised approaches trained on labeled egocentric data.The transfer uses labeled exocentric data and unlabeled synchronized exocentric-egocentric pairs.Source: §6 · p. 14

A Details on Feature Distillation

  • Feature distillation trains a student on synchronized Exo–Ego clips to match a frozen exocentric teacher’s TSM features by MSE, while a DINOv2 adapter is trained without fine-tuning DINOv2.The adapter is a residual transformer encoder that maps Ego features toward Exo features.Source: A Details on Feature Distillation · p. 19

B Details on TAS Model Distillation

  • TAS-model distillation uses fixed teacher and student C2F-TCNs on synchronized pairs, summing MSE alignment across temporal hierarchy levels; aligning all levels performed best.The teacher is trained on features from labeled exocentric videos.Source: B Details on TAS Model Distillation · p. 20

C Details on Assembly101 - View Selection

  • Assembly101 experiments use informative egocentric view e4 and two representative exocentric views, excluding ceiling-facing or redundant egocentric views.The dataset includes four monochrome first-person cameras and eight RGB third-person cameras.Source: C Details on Assembly101 - View… · pp. 20–21

D Training Details

The experiments use pretrained TSM models and large DINOv2 features, with separate optimization settings for model training and the transformer.

D Training Details

  • TSM models are pretrained on EPIC-KITCHENS-100, while large DINOv2 extracts per-frame features; TSM and C2F-TCN use ADAM with learning rate and weight decay 10^-4.Training uses early stopping with a maximum of 200 epochs and patience 20.Source: D Training Details · p. 22
  • The transformer is trained with SGD from learning rate 10^-2, and the main-paper experiments use the classification version of the TCC loss.The stated early-stop criterion halts training when performance on the training data ceases to improve.Source: D Training Details · p. 22

E Ablation on the use of Exo and Ego data to train the feature extractor on Assembly101

On Assembly101, adding egocentric training data markedly improves exocentric-to-egocentric adaptation, while exocentric data alone provides little benefit in the exocentric test setting. Distillation also brings results closer to the egocentric oracle.

E Ablation on the use of Exo and Ego data to train the feature extractor on Assembly101

  • In the exo-oracle setting, adding ego data changed edit score from 31.14 to 31.45 and MoF from 32.23 to 37.39, which the authors describe as little benefit.The passage attributes this pattern to a stronger signal from exocentric videos.Source: E Ablation on the use of Exo and Ego… · p. 22, Table 2 · p. 23
  • Adding exo data to the ego-oracle baseline raised edit score from 29.25 to 30.50 and MoF from 36.60 to 38.00.The authors suggest exocentric videos can regularize learning when testing on egocentric video.Source: E Ablation on the use of Exo and Ego… · p. 22
  • Edit score rose from 10.53 to 28.28, F150 from 0.56 to 16.81, and MoF from 5.42 to 35.37 when ego training data was added to the adaptation baseline.The caption identifies this as an egocentric-test comparison in Table 2.Source: E Ablation on the use of Exo and Ego… · p. 23, Table 2 · p. 23
  • TAS-model distillation achieved edit score 25.39 versus 29.25 and F150 15.52 versus 17.47 for the ego-oracle comparison, bringing performance closer to the oracle.The table caption characterizes feature plus TAS-model distillation as having a similar effect to using labeled ego data.Source: E Ablation on the use of Exo and Ego… · p. 23, Table 2 · p. 23

F Ablation on the effect of different feature extractors on Assembly101

On Assembly101, feature distillation brings exocentric-model performance close to egocentric-feature results. DINOv2 features also outperform exocentric TSM features in the egocentric evaluation setting.

F Ablation on the effect of different feature extractors on Assembly101

G Additional Qualitative Results on Assembly101

The Assembly101 qualitative examples compare segmentation outputs with ground truth and show how adaptation methods differ across DINOv2 examples.

G Additional Qualitative Results on Assembly101

  • Figures 5 and 6 show compared methods’ segmentations alongside ground truth, with colors identifying the action classes in each video.The supplied findings discuss DINOv2 examples from Figure 5.Source: G Additional Qualitative Results on… · p. 24

G.1 DINOv2

  • DINOv2 no-adaptation outputs are often over-segmented or under-segmented, while oracle baselines usually recover the action sequence but may miss temporal bounds.Over-segmentation appears in Figures 5c and 5d; under-segmentation appears in Figure 5h.Source: §G.1 · p. 24
  • Feature distillation often reduces over-segmentation without always recovering action order, while TAS model distillation predicts partially correct action sequences in several examples.Source: §G.1 · p. 24

G.2 TSM

The EgoExo4D supplementary results report Omnivore-feature accuracy, TSM qualitative outcomes, and evidence that distillation learns exo–ego correspondences. The benchmark setup selects a representative exocentric view and uses shared Omnivore features.

G.2 TSM

  • TSM no-adaptation often under-segments; feature distillation partially recovers action ordering on challenging clips and generally produces more accurate segmentations.Oracle baselines often recover partially correct action execution orders.Source: §G.2 · pp. 24–25

H Details on EgoExo4D - View Selection

  • The selected EgoExo4D exocentric view is the camera marked “best exo” for most segments, which shows more action detail than a generic camera.When best-exo annotations are unavailable, the dataset’s most frequently marked camera is selected; the views also vary across environments.Source: H Details on EgoExo4D - View Selection · p. 25, Figure 4 · p. 25

I Details on EgoExo4D - Omnivore Features

J Source of Improvement and role of synchronization

  • 20.52% of retrieved ego segments share the query’s action class with TSM-EGO features, versus 32.51% with distilled features, suggesting learned exo–ego correspondences.For each exo action segment, the experiment retrieves five nearest ego segments.Source: J Source of Improvement and role of… · p. 26

K Generalization to other tasks

The supplementary section reports that the distillation approach also benefits action recognition on Assembly101, with limited absolute performance from a simple model.

K Generalization to other tasks

Loading 2312.02638v3…