PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Modal Domain Adaptation for Fine-Grained Action Recognition

Jonathan Munro, Dima Damen

arXiv:2001.09691v2cs.CV

TL;DR

Fine-grained action datasets are environmentally biased, making transfer from labelled source environments to unlabelled target environments difficult. The paper proposes MM-SADA, which combines modality-correspondence self-supervision with per-modality adversarial alignment, and reports improved transfer across EPIC-Kitchens domains.

  • Problem

    Fine-grained action datasets often contain sequences from few environments, while prior video UDA methods have not explored video’s multi-modal nature for fine-grained action recognition.

  • Method

    MM-SADA jointly uses modality-correspondence self-supervision on source and unlabelled target data with adversarial alignment through a domain discriminator for each modality.

  • Results

    MM-SADA outperforms source-only generalisation and alternative domain-adaptation strategies, with the conclusion reporting a 4.8% improvement over nonadapted models.

  • Takeaways & Limitations

    Jointly aligning individual modalities while learning correspondence across source and target domains can improve action-recognition transfer to unlabelled environments.

  • Takeaways & Limitations

    Future work is needed to assess additional modalities such as audio and additional self-supervised tasks for adaptation.

Abstract

from arXiv · show

Fine-grained action recognition datasets exhibit environmental bias, where multiple video sequences are captured from a limited number of environments. Training a model in one environment and deploying in another results in a drop in performance due to an unavoidable domain shift. Unsupervised Domain Adaptation (UDA) approaches have frequently utilised adversarial training between the source and target domains. However, these approaches have not explored the multi-modal nature of video within each domain. In this work we exploit the correspondence of modalities as a self-supervised alignment approach for UDA in addition to adversarial alignment. We test our approach on three kitchens from our large-scale dataset, EPIC-Kitchens, using two modalities commonly employed for action recognition: RGB and Optical Flow. We show that multi-modal self-supervision alone improves the performance over source-only training by 2.4% on average. We then combine adversarial training with multi-modal self-supervision, showing that our approach outperforms other UDA methods by 3%.

1. Introduction

Fine-grained action datasets often contain videos from few environments, creating environmental bias and poor cross-environment generalisation. The paper introduces multi-modal self-supervision and adversarial alignment for UDA in fine-grained action recognition.

  • Motivation: Fine-grained action recognition distinguishes specific interactions such as cutting a tomato from coarse activities such as preparing a meal.Applications include assistive technologies in homes and industry, but collecting and annotating such actions is difficult.
  • Environmental bias: Natural-interaction datasets offer realistic videos, but many actions are collected in only a few environments, limiting representation generalisation.Long untrimmed sequences may contain several fine-grained actions from one or a few environments.
  • Research gap: Deep UDA for video has received limited attention, and prior approaches neither target fine-grained actions nor exploit video’s multi-modal structure.Existing approaches commonly treat video as RGB images, whereas self-supervised methods have used multiple modalities.
  • Contribution: The paper proposes MM-SADA, combining adversarial alignment with multi-modal self-supervision on source and unlabelled target data.The approach is presented as a strategy for adapting fine-grained action-recognition models to unlabelled target environments.
  • Evaluation: The evaluation tests the approach across three domains from EPIC-Kitchens and provides a benchmark against alternative domain-adaptation strategies.Compared methods include batch-based normalisation, distribution discrepancy minimisation, and classifier discrepancy.

2. Related Works

Prior work spans general UDA, self-supervised video representation learning, and action-recognition adaptation, but environmental UDA for fine-grained action recognition remains underexplored. This paper addresses that gap with multi-modal correspondence self-supervision combined with per-modality adversarial alignment.

  • General UDA: General UDA methods adapt neural networks by minimising discrepancies between source and target representations, with gradient reversal enabling adversarial domain alignment.The supplied related-work passage also references domain discriminators and gradient reversal as common mechanisms.
  • Multi-modal UDA: Multi-modal UDA has used separate adversarial discriminators, mid-level fusion, and cross-modality attention, showing advantages over single-modality adaptation in bird image retrieval.That prior setting involved image and audio modalities with similar robustness to domain shift.
  • Self-supervision: Self-supervised domain adaptation has used auxiliary image tasks and has sometimes replaced or benefited adversarial training, but prior examples use single images.This paper instead exploits multiple video modalities for adaptation.
  • Self-supervision: Video action-recognition self-supervision models temporal structure or modality correspondence, including frame ordering, arrow-of-time prediction, and audio-RGB synchronisation.The paper tests correspondence-based self-supervision approaches in its UDA framework.
  • Action-recognition adaptation: Earlier action-recognition adaptation largely addressed viewpoint changes, while modalities such as RGB, flow, and depth were aligned and evaluated independently.These methods focused on geometric camera transformations rather than broader environmental shifts.
  • Action-recognition adaptation: Environmental UDA for action recognition has received limited recent attention, with deep approaches applying gradient-reversal adversarial training to video architectures.Reported architectures include C3D, TRN, and combined approaches.
  • Research gap: EPIC-Kitchens benchmark reports show a 12-20% accuracy drop on novel versus seen kitchens, while no previous effort applied UDA to fine-grained action datasets.The paper presents the first multi-modal UDA approach for fine-grained action recognition.

3. Proposed Method

MM-SADA adapts fine-grained action-recognition models by combining within-modality adversarial alignment with multi-modal self-supervised correspondence learning. The alignment losses use source and target data, while supervised action classification uses labelled source data.

  • MM-SADA combines adversarial alignment with multi-modal self-supervision to adapt action-recognition models across domains.The approach uses both within-modal domain alignment and cross-modal correspondence learning.
  • Late fusion combines modalities by summing their prediction scores and backpropagating the classification error to all modality streams.The modality-specific classifiers and feature extractors are trained through the consensus classification loss.
  • The classification loss is source-only, whereas adversarial and correspondence alignment losses are optimised using both unlabelled source and target data.This separates supervised action learning from domain-alignment objectives.
  • Each modality has a separate domain discriminator that predicts whether features come from the source or target domain.Separate discriminators penalise modality-specific domain features and train each modality’s feature representation.
  • The self-supervised classifier predicts whether sampled modalities correspond to the same action, using positive same-action pairs and negative different-action pairs.Correspondence learning is optimised over both source and target domains without action labels.

4. Experiments and Results

Experiments evaluate MM-SADA across three EPIC-Kitchens domains, comparing multimodal self-supervision and adversarial alignment with source-only and alternative adaptation methods. MM-SADA improves target accuracy overall, remains generally stable during training, and benefits both RGB and Flow streams.

  • Experimental setup: Experiments use three EPIC-Kitchens environments—P01, P22, and P08—as domains, evaluating pairs of domains with the eight largest action classes.These classes represent 80% of the training action segments and have differing distributions between domains.
  • Comparison with baselines: 4.8% average improvement over source-only performance is reported for MM-SADA on target domains.Table 1 compares MM-SADA with different alignment approaches.
  • Comparison with baselines: MM-SADA outperforms batch-based normalization by 3.1%, classifier discrepancy by 3%, and discrepancy minimization by 3.5%.The improvement is consistent across domain pairs; MM-SADA improves over source-only by up to 7.5% in 5 of 6 cases.
  • Training behavior: MM-SADA maintains higher target accuracy than source-only training throughout training except for the D3 →D2 case.The training curves show minimal fluctuations, which matters because UDA cannot use target labels for early stopping.
  • Ablation study: 2.4% improvement over no adaptation comes from self-supervision alone, while adversarial alignment adds a further 2.4%.Compared with adversarial alignment alone, MM-SADA improves performance in 5 of 6 domains by up to 3.2%.
  • Ablation study: For D3 →D2, self-supervision alone exceeds source-only and other reported methods by 1.1%, while setting λd = 0.5 yields 56.9% accuracy.With λd = 1, the combined method does not beat the source-only baseline for this pair.
  • Individual modalities: MM-SADA improves RGB by 6.5% on average over source-only and Flow by 1.6%, with multimodal self-supervision adding 2.4% and 1.1% over adversarial alignment alone.Predictions are evaluated separately before late fusion, while both modalities remain available for self-supervision.

5. Conclusion and Future Work

The paper concludes that combining per-modality adversarial training with multi-modal self-supervision improves adaptation of fine-grained action recognition models to unlabelled environments. It identifies broader modality coverage and additional self-supervised tasks as future directions.

  • Conclusion: Multi-modal self-supervision predicts correspondence across modalities and serves as an effective domain adaptation method.It is jointly optimised over source and target domains.
  • Conclusion: Self-supervision alone can outperform domain alignment methods by jointly optimising the self-supervised task over both domains.
  • Conclusion: 4.8%: combining multi-modal self-supervision with adversarial training outperforms nonadapted models.
  • Conclusion: Aligning individual modalities while learning self-supervision on source and target domains improves transfer to unlabelled environments.
  • Future Work: Future work will add modalities such as audio and explore additional individually trained and multi-task self-supervised adaptation tasks.
Loading 2001.09691v2…