PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal Distillation for Egocentric Action Recognition

Gorjan Radevski, Dusan Grujicic, Marie-Francine Moens, Matthew Blaschko, Tinne Tuytelaars

arXiv:2307.07483v2cs.CV

TL;DR

Egocentric action recognition benefits from cues beyond RGB, but processing additional modalities at inference can make models costly to deploy. This paper distills a multimodal ensemble into an RGB student. The reported students are more accurate and better calibrated than comparable ground-truth-trained models, with reduced computational complexity.

  • Problem

    Additional modalities can improve egocentric action recognition, but their inference-time processing can be impractical under limited compute budgets.

    Source: §1 · p. 1

  • Method

    The method distills predictions from separately trained modality-specific teachers, combined with held-out-performance weights, into a student that uses RGB at inference.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Multimodal-teacher students are more accurate and better calibrated than architecturally equivalent models trained from scratch or omnivorously.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The approach offers a trade-off between the performance of a cumbersome multimodal ensemble and the lower computational complexity of unimodal inference.

    Source: §5 · p. 8

    The study considers only action recognition and a limited set of modalities.

    Source: §5 · pp. 8–9

Abstract

from arXiv · show

The focal point of egocentric video understanding is modelling hand-object interactions. Standard models, e.g. CNNs or Vision Transformers, which receive RGB frames as input perform well. However, their performance improves further by employing additional input modalities that provide complementary cues, such as object detections, optical flow, audio, etc. The added complexity of the modality-specific modules, on the other hand, makes these models impractical for deployment. The goal of this work is to retain the performance of such a multimodal approach, while using only the RGB frames as input at inference time. We demonstrate that for egocentric action recognition on the Epic-Kitchens and the Something-Something datasets, students which are taught by multimodal teachers tend to be more accurate and better calibrated than architecturally equivalent models trained on ground truth labels in a unimodal or multimodal fashion. We further adopt a principled multimodal knowledge distillation framework, allowing us to deal with issues which occur when applying multimodal knowledge distillation in a naive manner. Lastly, we demonstrate the achieved reduction in computational complexity, and show that our approach maintains higher performance with the reduction of the number of input views. We release our code at https://github.com/gorjanradevski/multimodal-distillation.

1. Introduction

Egocentric action recognition centers on hand-object interactions, but clutter, occlusion, and motion blur can make RGB-only understanding challenging. The paper proposes using multimodal information during training while retaining RGB-only inference, aiming to preserve performance without deployment-time modality costs.

1. Introduction

  • Egocentric scenes can obscure hands or objects and include brief interactions and motion blur, challenging interpretation from RGB frames alone.The paper identifies hand-object interactions as the focal point of egocentric vision.Source: §1 · p. 1
  • Additional modalities can improve recognition, but their inference-time processing raises memory and compute demands, limiting practical deployment.The paper gives embedded devices as an example of a limited-compute setting.Source: §1 · pp. 1–2
  • The proposed route transfers knowledge from a high-performing multimodal ensemble to a standard RGB-based action-recognition student.The ensemble is described as unviable for deployment, while the student uses RGB at inference.Source: §1 · pp. 1–2
  • The paper reports that students taught by multimodal teachers are more accurate and better calibrated than architecturally equivalent models trained from scratch or omnivorously.It also presents a principled distillation approach and reports competitive performance with lower computational complexity.Source: §1 · p. 2

2. Related Work

Prior work uses distillation across modalities and multiple modalities at inference, while this paper targets settings where additional modalities are available only during training. It distinguishes its approach from feature imitation and models designed to handle missing or interchangeable modalities.

2. Related Work

  • The paper situates its method within knowledge distillation from ensembles, transferring knowledge from a multimodal teacher ensemble to a student.The cited ensemble-distillation line transfers knowledge from large teachers to lightweight students.Source: §2 · p. 2
  • Unlike cross-modal distillation and multimodal knowledge expansion, this work uses multimodal data during training because obtaining additional modalities at inference may be prohibitive.Prior cross-modal examples use RGB teachers with depth, optical flow, or audio students; multimodal expansion instead trains a multimodal student from a unimodal teacher.Source: §2 · p. 2
  • Earlier action-recognition distillation uses a hallucination stream to imitate another modality’s intermediate features, whereas this work applies standard distillation across several modalities.The cited earlier approach reports gains over an RGB baseline using RGB and depth.Source: §2 · p. 2
  • Multimodal video methods often rely on additional inference-time inputs and processing, including object detectors and trackers; this paper instead distills from those modalities into an RGB-inference model.The related work notes that added processing may be prohibitive under limited compute budgets.Source: §2 · p. 2
  • Omnivorous models process different modalities interchangeably, while this work compares an architecturally matched student distilled from a multimodal teacher against an Omnivorous variant.The paper reports that the distilled student generalizes better than that variant.Source: §2 · p. 2

3. Methodology

The methodology first formulates egocentric action recognition as predicting action classes from a sequence, then distills predictions from modality-specific teachers into an RGB student. The teacher ensemble is weighted using held-out performance, and the student combines distillation and labeled-data losses.

3.1. Egocentric Action Recognition

  • The method trains action classifiers on egocentric sequences with cross-entropy, using separate prediction heads and summed losses for noun-verb compositions.The input is a time sequence, and the output is a probability distribution over action classes or their noun and verb components.Source: §3.1 · pp. 2–3

3.2. Multimodal Knowledge Distillation

  • Each modality has a separately trained teacher, whose predictions are combined to teach a single student that relies on RGB inputs alone.The approach adapts standard knowledge distillation by making the teacher an ensemble of models that receive distinct modalities.Source: §3.2 · p. 3
  • Teacher weights are softmax-normalized negative cross-entropies estimated on held-out samples, with a temperature parameter controlling the weights’ entropy.The weighted teacher predictions are then aggregated into the ensemble output.Source: §3.2 · p. 3
  • Student training combines KL divergence from teacher probabilities with the action-recognition cross-entropy loss, balanced by λ; τ adjusts prediction entropy and rescales the KL term.At λ = 0 the objective is RGB-model training, while λ = 1 uses distillation alone.Source: §3.2 · p. 3
  • The paper argues that the multimodal ensemble is a better approximation of Bayes class probabilities than a single-modality model, which may improve student generalization through a lower-variance objective.The preliminary comparison reports higher accuracy and lower calibration error for the ensemble at γ = 1 and γ = 30.0.Source: §3.2 · pp. 3–4

4. Experiments & Discussion

The experiments evaluate multimodal distillation for egocentric action recognition, including recognition, generalization, calibration, and efficiency. They also compare the approach with Omnivorous models and examine teacher weighting and qualitative predictions.

4.1.1 Recognizing Egocentric Actions

  • On Something-Something, RGB+OF+OBJ distillation improves top-1 accuracy over the RGB baseline by 3.7%; on Epic-Kitchens, RGB+OF and RGB+OF+A students outperform the baseline and modality-specific models.On Something-Something, students are nearly on par with the teacher in top-5 accuracy; Epic-Kitchens students distilled from RGB+A fall below the baseline.Source: §4.1.1 · p. 5, Table 2 · p. 6

4.1.2 Generalizing to Unseen Environments & Objects

  • On Something-Else, the RGB+OF+OBJ student exceeds the RGB baseline by 7.3% in top-1 accuracy, and students generally outperform the baseline across unseen-setting evaluations.The evaluations use Something-Else and the Epic-Kitchens Unseen split.Source: §4.1.2 · p. 5, Table 3 · p. 6

4.1.3 Comparison with Omnivorous Models

  • Distillation achieves higher performance than the Omnivore model, although Omnivore improves over the baseline by about 5.5% on Something-Else; both approaches are viable for RGB-only inference.The compared models use the same Swin-T architecture and infer from RGB frames.Source: §4.1.3 · pp. 5–6, Table 4 · p. 6

4.1.4 Effect on Model Calibration

  • Across datasets, distillation improves student calibration, and using more modalities in the teacher ensemble further improves calibration.Calibration is evaluated with Expected Calibration Error (ECE).Source: §4.1.4 · p. 6, Figure 3 · p. 6

4.1.5 Per-Class Performance Breakdown

  • On Something-Something, distillation improves accuracy for all 20 most frequent action classes; it generally improves performance across frequent actions on both datasets.Qualitative examples show the student picking up relevant cues from modalities and accurately predicting the action of interest.Source: §4.1.5 · pp. 6–7, Figure 4 · p. 7, §4.4 · p. 8

4.2. Ensemble Teacher Weighting

  • The best Epic-Kitchens result uses both task and distillation losses (λ = 0.8) and weights teachers by performance on 1,000 held-out videos with γ = 1.0.Lowering γ to 0.33 or raising it to 3.0 reduces performance; γ = 30.0, which effectively equalizes weights, also performs well.Source: §4.2 · p. 7, Table 5 · p. 7

4.3. Efficiency Analysis

  • The three-model Epic-Kitchens teacher ensemble requires 584.36 GFLOPs, while a distilled Swin-T student requires 140.33 GFLOPs, a 75.98% reduction.Students also outperform their ground-truth-trained counterparts, and the best student is comparable to the more expensive RGB+OF+A teacher.Source: §4.3 · p. 7, Figure 5 · p. 8

4.3.1 Effect of Test-Time Augmentation

  • With fewer temporal clips or spatial crops at inference, the distilled model is less adversely affected than the ground-truth-trained model.This comparison uses λ = 1.0 for the distilled model and λ = 0.0 for the ground-truth-trained model.Source: §4.3.1 · p. 8, Figure 6 · p. 8

5. Conclusion

The paper uses multimodal data during training to improve RGB-only action recognition, trading off the performance of a cumbersome multimodal ensemble against lower inference complexity. Its evaluation covers action recognition and a limited set of modalities.

5. Conclusion

  • The distillation approach improves RGB-only models over models trained on ground-truth labels while balancing multimodal-ensemble performance against lower inference complexity.It also relies less on expensive test-time augmentations.Source: §5 · p. 8
  • The study evaluates multimodal distillation only for action recognition, while noting it could be applied to other egocentric tasks.Source: §5 · p. 8
  • The experiments use a limited set of modalities; depth, hand poses, and inertial-sensor motion are suggested for future work.Source: §5 · p. 9

Supplementary material

The supplementary material provides dataset and modality details, additional training-method information, and expanded experimental analyses.

Supplementary material

  • The supplement extends discussion of the datasets and describes the modalities and modality-specific models used.Source: Supplementary material · p. 12
  • It includes an ablation study on Something-Something and Something-Else, alongside details of approaches that leverage multimodal training data.Source: Supplementary material · p. 12
  • Additional analyses include learning curves and per-class performance breakdowns for Epic-Kitchens and Something-Something.Source: Supplementary material · p. 12
  • The supplement also provides additional qualitative examples from Epic-Kitchens and Something-Something.Source: Supplementary material · p. 12

A. Datasets

The experiments use EPIC-Kitchens, a benchmark of egocentric kitchen activities with compositional action labels and RGB, audio, and optical-flow inputs.

A. Datasets

  • EPIC-Kitchens contains 700 videos from 32 participants, totaling 100 hours of egocentric kitchen activity footage.Source: §A.1 · p. 12
  • Its action labels decompose into nouns for active objects and verbs for activities, with 300 noun and 97 verb categories.Source: §A.1 · p. 12
  • The experiments use RGB frames, audio extracted from the videos, and optical flow released by the dataset authors.Source: §A.1 · p. 12

A.1.1 Epic-Kitchens Unseen Participants

The supplementary dataset descriptions cover an unseen-participant Epic-Kitchens split, Something-Something, and the Something-Else compositional-generalization split.

A.1.1 Epic-Kitchens Unseen Participants

  • The Epic-Kitchens unseen-participant split contains 1,065 action sequences from two participants absent from training, testing student robustness to this distribution shift.Source: §A.1.1 · p. 12

A.2. Something-Something

  • Something-Something V2 has 174 object-agnostic hand-action classes, placing increased emphasis on the temporal relationships that characterize actions.Source: §A.2 · p. 12

A.3. Something-Else

  • Something-Else separates training and test objects to assess compositional generalization; prior work reports lower RGB-model performance there than on standard Something-Something.Source: §A.3 · p. 12

B.1. RGB frames (RGB)

The appendix specifies how RGB, optical flow, audio, and object detections are prepared and encoded for the experiments.

B.1. RGB frames (RGB)

  • RGB uses the Swin-T backbone and standard dataset setting, with random resized crops and augmentation for training and a central crop at inference.For R3D models, the crop size is reduced to 112.Source: §B.1 · pp. 12–13

B.2. Optical Flow (OF)

  • Optical flow uses the RGB processing setup and Swin-T, with the two velocity components plus a zero-valued channel as input.It uses RGB augmentations except color jitter.Source: §B.2 · p. 13

B.3. Audio (A)

  • Epic-Kitchens audio is converted to 16 kHz mono and represented as 128-bin mel spectrograms with 224 timesteps, then augmented and fed to Swin-T.Training uses time and frequency masking; inference uses a central crop and repeats the spectrogram across three input channels.Source: §B.3 · p. 13

B.4. Object Detections (OBJ)

  • Object detections are represented by bounding boxes and categories, then encoded by STLT’s spatial and temporal Transformers.The spatial Transformer encodes relations within each frame; the temporal Transformer encodes relations across frame embeddings.Source: §B.4 · p. 13

C. Loss Term Weighting - Something-Something & Something-Else

The Something-Something and Something-Else ablation examines loss weighting and finds that ground-truth supervision adds little when modality-specific models are strong and complementary.

C. Loss Term Weighting - Something-Something & Something-Else

  • On Something-Something and Something-Else, adding ground-truth cross-entropy to distillation has a small effect because modality-specific models perform well and complement one another.The authors therefore report less need for joint training with both ground-truth labels and distillation loss on these datasets.Source: C. Loss Term Weighting -… · p. 13
  • The preceding Table 5 findings indicate that ground-truth cross-entropy with distillation addresses inferior modality-specific teachers, while weighting ensemble teachers improves performance further.Teacher weights minimize each teacher’s cross-entropy on a 1,000-sample holdout set.Source: C. Loss Term Weighting -… · p. 13
  • Table 6 reports the Something-Something and Something-Else ablation, with λ balancing distillation and cross-entropy losses and γ denoting ensemble-teacher weighting temperature.Source: Table 6 · p. 14

D. Details on action recognition models trained on multimodal data

The appendix compares approaches that use multimodal training with RGB-only inference, reports learning curves and class-level results, and adds qualitative examples.

D. Details on action recognition models trained on multimodal data

  • ModDrop trains with randomly dropped modalities, while DMCL selects a teacher per sample and distills its soft labels; this work instead uses standard knowledge distillation.The cited DMCL framework is tested on non-egocentric data; the paper describes its own approach as flexible to ensemble additions and student retraining with fixed existing models.Source: D. Details on action recognition… · p. 14
  • For its Omnivore baseline, the paper uses multimodal downstream training; homogeneous randomly selected modality batches perform better than mixed-modality batches, which resemble RGB-only training.Omnivore uses multimodal pretraining and RGB fine-tuning for downstream inference in the cited work.Source: D. Details on action recognition… · p. 14

E. Learning curves

  • The learning curves plot top-1 validation accuracy by epoch and show the distilled student converging to better generalization than ground-truth-only training on both datasets.Source: E. Learning curves · p. 14, Figure 8 · p. 15

F. Per-Class Performance Breakdown

G. Qualitative Examples

Related papers

Loading 2307.07483v2…