PaperlayerView exact revision on arXiv

Source-linked AI summary

EgoDistill: Egocentric Head Motion Distillation for Efficient Video Understanding

Shuhan Tan, Tushar Nagarajan, Kristen Grauman

arXiv:2301.02217v1cs.CV

TL;DR

Dense egocentric video models are computationally expensive, limiting their practicality for resource-constrained or real-time settings. EgoDistill reconstructs dense video features from sparse frames and IMU head motion through distillation, and reports improved accuracy-efficiency trade-offs on Ego4D and EPIC-Kitchens.

  • Problem

    Dense clip-based egocentric models process many frames with temporally-aware operations, making them computationally heavy for resource-constrained AR/VR and real-time applications.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    EgoDistill distills an expensive video model’s clip features into a lightweight model using sparse RGB frames and IMU head motion, with self-supervised IMU feature learning.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    Across Ego4D and EPIC-Kitchens, EgoDistill outperforms state-of-the-art efficient methods; with 1–4 frames, it uses 4× to 8× less computation than adaptive-sampling approaches.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that EgoDistill can accelerate egocentric video understanding using a data stream already ubiquitous in egocentric cameras.

    Source: §5 · p. 8

    Modifying hyperparameters to control the accuracy-efficiency trade-off results in unstable training and unreliable performance.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Recent advances in egocentric video understanding models are promising, but their heavy computational expense is a barrier for many real-world applications. To address this challenge, we propose EgoDistill, a distillation-based approach that learns to reconstruct heavy egocentric video clip features by combining the semantics from a sparse set of video frames with the head motion from lightweight IMU readings. We further devise a novel self-supervised training strategy for IMU feature learning. Our method leads to significant improvements in efficiency, requiring 200x fewer GFLOPs than equivalent video models. We demonstrate its effectiveness on the Ego4D and EPICKitchens datasets, where our method outperforms state-of-the-art efficient video understanding methods.

1. Introduction

Dense egocentric video models are computationally heavy for resource-constrained AR/VR and real-time use. EgoDistill combines sparse frames with IMU head motion to reconstruct dense video features, aiming to improve efficiency while preserving recognition performance.

1. Introduction

  • Dense, temporally-aware clip models are computationally heavy, making them impractical for resource-constrained AR/VR devices and real-time applications.The paper frames efficient egocentric video understanding as an important unsolved problem.Source: §1 · p. 1
  • The approach treats sparse-frame appearance as a source of scene semantics and head motion as a source of dynamics across neighboring frames.The authors hypothesize that for short clips, a single frame can convey much of the semantic content while head motion reveals how appearance changes.Source: §1 · pp. 1–2
  • EgoDistill reconstructs dense clip features from as few as one RGB frame plus IMU head motion, using distillation from an expensive video model.The lightweight model is trained to approximate the representation extracted by the video model and used directly for recognition at inference.Source: §1 · p. 2
  • 200× fewer GFLOPs than the original video model; EgoDistill also reduces inference for a typical 50-minute video from 25 minutes to 36 seconds.The 200× figure compares computation with the original video model; the timing example is reported separately for a typical 50-minute video.Source: §1 · p. 2
  • 4× to 8× less computation with 1–4 frames, while surpassing state-of-the-art adaptive-sampling methods by a large accuracy margin.The comparison is reported on Ego4D and EPIC-Kitchens.Source: §1 · p. 2

2. Related Work

Related work uses IMU for activity recognition, odometry, and visual-feature learning, while efficient video methods reduce computation through lightweight models or adaptive content selection. EgoDistill distinguishes itself by using head motion to infer expensive egocentric video features from sparse visual input.

2. Related Work

  • Activity-recognition models use body-, waist-, or pocket-mounted IMUs, whereas EgoDistill uses head motion to reconstruct video features.Prior video work also couples RGB with audio [22] [38], optical flow, or both [32] [36]; relatively fewer works use IMU [48].Source: §2 · p. 2
  • Inertial odometry estimates camera position and orientation, while IMU-guided visual-feature learning pursues other objectives [14].EgoDistill instead uses head motion to infer pretrained video features for efficient video understanding.Source: §2 · p. 2
  • Adaptive-selection methods skip clips, frames, channels, regions, or tokens [39] [24] [49] [50] [65]; EgoDistill instead infers full clip features from sparse images and motion.The paper describes the approaches as complementary.Source: §2 · p. 2
  • Multimodal distillation transfers knowledge across modalities [23] [28] [45] [21]; ListenToLook [22] combines clip subsampling and video-to-audio distillation, unlike EgoDistill’s head-motion/RGB approach.The comparison is framed for egocentric video understanding.Source: §2 · p. 3

3. Approach

EgoDistill uses sparsely sampled frames and head motion from IMU to approximate heavy video-model features. Its approach combines lightweight multimodal inference, teacher–student distillation, and self-supervised IMU feature pretraining.

3.1. Egocentric action recognition

  • Action recognition assigns each fixed-length clip scores over action classes, while clip-based models process many frames at high computational cost.The stated goal is a frame-based model that approximates clip-based performance while remaining efficient.Source: §3.1 · p. 3, §3.2 · p. 3

3.2. Efficient video inference with head motion

  • Sparse RGB frames provide appearance and six-dimensional accelerometer and gyroscope readings provide camera-motion information for a lightweight model.The model uses 1–4 uniformly sampled frames; the paper also notes that learned features can encode predictable scene motion tied to visual content.Source: §3.2 · p. 3, §3.3 · p. 4

3.3. Video feature distillation with IMU

  • EgoDistill fuses image and IMU features in a two-stream model; for multiple input frames, a GRU aggregates image features over time.A fused representation feeds a classifier, while the broader pipeline trains the lightweight model to approximate a heavy video model.Source: §3.3 · p. 3, §3.2 · p. 3, Figure 2 · p. 4
  • Training combines feature matching, teacher–student knowledge distillation, and ground-truth action classification losses.The distillation loss uses the heavy model’s class-logit distribution as a soft target; α balances distillation and activity training, while β weights feature matching.Source: §3.3 · p. 4

3.4. Self-supervised IMU feature learning

  • Self-supervised pretraining uses clip start and end frames to predict an IMU feature, encouraging motion features associated with visual changes, including scene motion.The pretraining objective combines L1 and NCE losses before the main training stage.Source: §3.4 · p. 5

4. Experiments

Experiments evaluate EgoDistill for egocentric action recognition on Ego4D and EPIC-Kitchens, comparing sparse-frame and IMU distillation with efficient video baselines and analyzing its components and behavior.

4.1. Experimental setup

  • The experiments construct 94-class Ego4D (8.5k training, 3.6k evaluation videos) and 62-class EPIC-Kitchens (29k training, 6.2k evaluation videos) datasets, using verb labels.Source: §4.1 · p. 5

4.3. Main Results

  • With N=4 frames, EgoDistill surpasses STTS by 7.4% and AdaFuse by 4.2% on EPIC-Kitchens with 2× fewer GFLOPs, and both by 2.1% on Ego4D.Figure 3 compares accuracy and efficiency on both datasets.Source: Figure 3 · p. 6, §4.3 · pp. 6–7

4.2. Baselines

  • Comparisons include image-only and IMU-removed distillation baselines, adaptive video sampling, token selection, and audio-based distillation.AdaFuse and STTS use four input frames for comparison; ListenToLook varies input frames to match EgoDistill’s efficiency-accuracy trade-off.Source: §4.2 · pp. 5–6

4.4. Analysis

  • Across datasets and teacher architectures, EgoDistill consistently improves over VisOnly-Distill; better video teachers yield better student performance.Source: Table 2 · p. 6, §4.4 · p. 7

4.3. Main Results

  • At N=1 on EPIC-Kitchens, EgoDistill improves over VisOnly-Distill by 8.4% and VisIMU by 9.9%.The authors attribute the comparisons to the value of IMU context and video-model distillation, respectively.Source: §4.3 · p. 6

4.4. Analysis

  • IMU helps most for actions with predictable head motion, while small or unrelated motion and drastic, blurry movement are less useful for reconstruction.Examples of helpful actions include break, cut, and close; Figure 4 compares per-class accuracy gains, and Figure 5 shows best and worst reconstructed clips.Source: Figure 4 · p. 7, §4.4 · p. 7, Figure 5 · p. 7
  • Nearly 200× fewer GFLOPs than MotionFormer; EgoDistill is 6.5× faster than STTS while achieving 4.4% higher accuracy on EPIC-Kitchens.Source: Table 3 · p. 7, §4.4 · pp. 7–8

4.5. Qualitative Results

  • IMU-conditioned features preserve semantics and motion; on ambiguous close and put classes, gains over VisOnly-Distill are 20.3% and 10.4% on EPIC-Kitchens, and 8.5% and 3.9% on Ego4D.Examples show the features disambiguating close versus open and put versus take.Source: §4.5 · p. 8, Figure 6 · p. 8, Figure 7 · p. 8

5. Conclusion

EgoDistill approximates egocentric video features for fast action recognition and achieves a good accuracy–efficiency balance on Ego4D and EPIC-Kitchens. The conclusion identifies longer-term activity understanding and multi-view learning as future directions.

5. Conclusion

  • EgoDistill achieves a good accuracy–efficiency balance and outperforms state-of-the-art efficient video understanding methods on Ego4D and EPIC-Kitchens action recognition.Source: §5 · p. 8
  • EgoDistill is the first model presented to explore egocentric video feature approximation for fast recognition.Source: §5 · p. 8
  • Future work will investigate long-term activity understanding with room context and visual correspondence learning for multi-view videos.Source: §5 · p. 8

A. Supplementary Video

The supplementary video presents animated examples of EgoDistill’s reconstructed clips, retrieval, and scene-motion anticipation. The animations are recommended for viewing head motion and video dynamics more clearly.

A. Supplementary Video

B. Dataset Details.

The experiments use Ego4D and EPIC-Kitchens action-recognition data, with dataset-specific annotation choices and train/evaluation splits. Ego4D action recognition is constructed from Hands+Objects temporal-localization annotations, while EPIC-Kitchens uses its action-recognition benchmark.

B. Dataset Details.

  • Ego4D action clips run from each Hands+Objects annotation’s pre-frame to post-frame, with the annotated verb serving as the label.Source: B. Dataset Details · p. 12
  • The Ego4D subset contains 94 action classes and 12.1k videos, split into 8.5k training and 3.6k evaluation videos.Source: B. Dataset Details · p. 12
  • EPIC-Kitchens data uses annotations from its action-recognition benchmark and includes 100 hours of kitchen activity video.Source: B. Dataset Details · p. 12
  • The EPIC-Kitchens split is by camera-wearer, with P30, P35, and P37 videos assigned to evaluation, yielding 29k training and 6.2k evaluation videos across 62 classes.Source: B. Dataset Details · p. 12

C. Implementation Details.

Implementation uses per-clip mean-centering for IMU readings, specified image and IMU backbones, and two-stage training. For four-frame EgoDistill, uniform frame selection performs better on average than the compared heuristics.

C. Implementation Details.

  • Each IMU dimension is mean-centered per clip to address drifting and bias, leaving the model to focus on temporal motion patterns.Source: C. Implementation Details · p. 12
  • For ListenToLook audio input, the audio is subsampled at 16 kHz and processed with an STFT using a 400-sample Hann window and 160-sample hop length.Source: C. Implementation Details · p. 12
  • The image backbone is ImageNet-pretrained ResNet-18, the IMU backbone is a five-layer 1D Dilated CNN, and fusion uses concatenation and two fully connected layers.Source: C. Implementation Details · p. 12
  • Training first self-supervises the IMU encoder, predictor, and fusion network with a fixed pretrained image encoder, then initializes and fine-tunes the model for video-feature distillation.Source: C. Implementation Details · p. 12
  • Uniform frame selection achieves better results on average than the compared heuristics for EgoDistill with N = 4.Source: Table 4 · p. 13

D. Analysis.

The analysis compares frame-selection heuristics and explains why EgoDistill uses only a few frames. Uniform sampling performs best on average at N = 4, while the small-frame setting targets applications where loading more frames costs time and memory.

D. Analysis.

  • At N = 4, uniform sampling performs best on average versus random, first, last, and center-frame selection.The authors suggest it provides broader coverage of semantic contexts and scene motion.Source: D. Analysis · p. 13
  • EgoDistill uses N = 1 to 4; its results are not directly comparable to video clip models using dense frames.Larger N, such as 8 or 16, could improve all methods, but with more computational cost.Source: D. Analysis · p. 13
  • Using only a few frames can benefit streaming video and low-resource AR/VR devices because loading additional frames takes time and memory.Source: D. Analysis · p. 13
Loading 2301.02217v1…