PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Scale Temporal Domain Alignment for Federated Video Domain Adaptation

Lee En-Yi Hannah, Haozhi Cao, Yuecong Xu

arXiv:2608.29186v1cs.CV

TL;DR

FVDA lacks established video-domain alignment methods despite the need to transfer knowledge across heterogeneous, privacy-preserving source and target domains with temporal variation. METAL uses multi-scale temporal representations, per-scale knowledge distillation and voting, scale coordination, and late fusion using only model parameters. It outperforms current FDA methods across Epic-Kitchens-55 and Daily-DA, with gains up to 28.47%.

  • Problem

    FVDA is under-explored because video adaptation must address temporal variation within videos and across heterogeneous source domains without cross-domain data access.

  • Method

    METAL learns multi-scale temporal representations, performs per-scale knowledge voting and distillation, applies L2 variance scale coordination, and uses late fusion with parameter-only communication.

  • Results

    28.47% relative performance gain is achieved over current FDA methods, with METAL outperforming all current FDA methods on Epic-Kitchens-55 and generally improving performance on Daily-DA.

  • Takeaways & Limitations

    Multi-scale distillation, scale coordination, and late fusion provide a supported framework for transferring distinct temporal information across federated video domains.

  • Takeaways & Limitations

    The setting assumes unlabeled target videos, shared classes between source and target domains, and communication restricted to model parameters.

Abstract

from arXiv · show

Federated Video Domain Adaptation (FVDA) enables collaborative learning across distributed and non-IID video datasets while preserving privacy, but is under-explored due to challenges in aligning temporal information. We propose Multi-scalE Temporal domAin aLignment (METAL), a novel framework that leverages temporal information at multiple resolutions to improve cross-domain video action recognition with only model parameter transfers. METAL trains per-scale transformer encoders on source-clients, then performs independent knowledge voting at each temporal scale to generate robust pseudo-labels on the target-server. A novel $L_2$ variance penalty enforces cross-scale consistency during scale-based knowledge distillation, preventing a singular dominant scale. The late fusion aggregates features across different scales, where the fusion head is trained via knowledge distillation using confidence-weighted aggregation of scale-wise predictions, enabling the model to effectively exploit complementary temporal information for final predictions. Experiments on Epic-Kitchens-55 and Daily-DA demonstrate state-of-the-art performances, with gains up to 28.47% over current FDA methods. Ablation studies prove that multi-scale distillation and scale coordination are critical for effective temporal knowledge transfer.

1. Introduction

FVDA extends privacy-preserving domain adaptation to video, where temporal variation within videos and across heterogeneous source clients complicates alignment. METAL addresses this gap with multi-scale temporal modeling, per-scale knowledge transfer, scale coordination, and late fusion, achieving substantial gains over existing FDA methods.

  • FDA transfers knowledge from statistically heterogeneous source-clients to a target-server without sharing data, but source models can generalize poorly without target-domain alignment.
  • Video domain adaptation must align both spatial and temporal dimensions, while existing FDA methods are primarily image-based.
  • Temporal characteristics vary within a video, so effective alignment must represent multiple motion frequencies simultaneously.
  • FVDA further requires combining information from multiple statistically distinct source-clients without cross-client data access.
  • METAL learns multi-temporal-scale representations, performs per-scale knowledge distillation using generalized source models, coordinates scales with L2 variance minimization, and applies late fusion.
  • 28.47% relative performance gain is achieved over current FDA methods on cross-domain action-recognition datasets including Epic-Kitchens-55 and Daily-DA.

2. Related Works

Prior FDA research uses invariant features, adaptation losses, prototypes, knowledge distillation, adversarial training, clustering, and pseudo-labeling, but video settings add temporal-domain alignment challenges.

  • FDA methods pursue domain-invariant features through adaptation losses, prototype or gradient aggregation, knowledge distillation, and adversarial training.
  • Adversarial methods can handle larger domain shifts but are computationally expensive and sensitive to features such as backgrounds and lighting.
  • Knowledge distillation is lightweight but may lack granularity, while prototype or gradient aggregation raises privacy concerns and depends on good feature extractors.
  • Unsupervised FDA combines clustering or source-model pseudo-labels with domain-adaptation techniques when target data are unlabeled.
  • Video UDA additionally requires alignment of temporal domains, beyond the domain-shift problem addressed in image-based adaptation.

3. Proposed Method

METAL considers federated adaptation from multiple labeled source domains to an unlabeled target domain under parameter-only communication. Its architecture performs per-scale source voting, target distillation, scale coordination, fusion, and federated updates.

  • FVDA is formulated as a multi-source-single-target problem with source clients and a target server.
  • At each temporal scale, source predictions are knowledge-voted into high-confidence pseudo-labels that supervise corresponding target students through per-scale KL divergence.
  • An L2 variance penalty enforces balanced knowledge transfer across scales, while a fusion head distills confidence-weighted ensemble predictions from concatenated multi-scale features.
  • Federated averaging updates the models after each communication round using consensus-based domain weights.
  • Each source domain contains labeled videos, whereas the target domain contains unlabelled videos with unobserved labels.
  • Source and target domains have different distributions but share the same C classes, and source data remain inaccessible during adaptation.

3.1. Multi-Scale Temporal Feature Representation

METAL represents each action segment as a sequence of frame-features and constructs multiple temporal views by sampling at different frequencies. These views capture motion patterns from fine-grained to coarse levels and support robustness to temporal variation across domains.

  • An action segment is represented as a temporal sequence of frame-features, with each feature constructed from a centered clip to preserve temporal information.
  • For each temporal scale, METAL uniformly samples T/s frames from the original sequence to construct multi-scale representations.
  • The resulting temporal views capture motion patterns ranging from fine-grained to coarse-level action dynamics.
  • Multi-scale temporal sampling enables scale-invariant features that are robust to temporal variations across domains.

3.2. Source Model Training

METAL uses independent transformer encoders at each temporal scale to extract video representations and classifiers to predict action labels. Source clients train these scale-specific models independently with cross-entropy and upload only their parameters.

  • Per-Scale Encoder Architecture: Each temporal scale uses a transformer encoder to map sampled frame features into a representation for action classification.The encoder applies a multi-layer transformer and layer normalization with a CLS token before classification.
  • Source Training Objective: Each source client independently trains its scale-specific encoder and classifier using the standard cross-entropy loss.The loss compares one-hot labels with predicted class probabilities for each scale.
  • Source Training Objective: After training, each source client uploads the parameters of its three trained scale-specific models to the target server.Communication transfers model parameters rather than source video data.

3.3. Per-Scale Knowledge Distillation

Per-scale knowledge distillation creates target-domain teachers and students independently at each temporal resolution. Reliable consensus pseudo-labels guide student training, while an L2 variance penalty coordinates learning across scales to prevent dominance by one scale.

  • Per-Scale Generalised Source Model Creation: METAL performs separate knowledge voting at each temporal scale because source-domain consensus can differ across scales.Source predictions are collected into a per-scale knowledge matrix, then majority voting identifies consensus labels.
  • Per-Scale Generalised Source Model Creation: Samples whose per-scale consensus confidence exceeds threshold τ are retained as reliable pseudo-labels.The confidence gate is linearly annealed to avoid eliminating too many domains early in training.
  • Per-Scale Knowledge Distillation Loss: For each scale, a student minimizes KL divergence between its predictions and the corresponding consensus pseudo-label, weighted by sample reliability.The binary reliability weight indicates whether a sample passes the confidence threshold.
  • Scale Coordination via L2 Variance Minimisation: The L2 variance penalty minimizes differences among per-scale KL losses to encourage balanced knowledge transfer across temporal resolutions.The square root keeps the penalty on the same scale as the KL losses and prevents any single scale from dominating.
  • Scale Coordination via L2 Variance Minimisation: The total adaptation loss combines the per-scale KL losses with β-weighted scale coordination, with gradients flowing through all scale models simultaneously.The variance term connects scale updates during backpropagation.

3.4. Scale Fusion

METAL combines independently learned temporal representations through late fusion. A separate fusion classifier is trained with confidence-weighted ensemble pseudo-labels, while consensus focus weights more informative source domains during aggregation.

  • Late Fusion: Late fusion concatenates features from all temporal scales after scale-specific student training.Independent learning reduces overlapping information that could arise from earlier lateral connections.
  • Fusion Distillation: The fusion classifier is trained by distilling an ensemble pseudo-label formed from confidence-weighted predictions across scales.A union strategy allows confident predictions from any temporal scale to contribute.
  • Fusion Distillation: Fusion-loss samples are weighted by the maximum scale confidence, indicating whether at least one scale provides reliable consensus.The fusion classifier produces predictions from the fused feature representation.
  • Domain Weighting: Consensus focus assigns source-domain weights according to how often each domain agrees with the final consensus prediction.The scores are normalized before weighting domains.
  • Federated Aggregation: Federated averaging aggregates encoder, classifier, and fusion-classifier parameters after each communication round while preserving parameter-only communication.More informative domains contribute more to the target model through their aggregation weights.

4. Experiments

METAL is evaluated on Epic-Kitchens-55 and Daily-DA under federated domain adaptation, using top-1 target accuracy and comparisons with current FDA methods. The experiments also include sensitivity, ablation, and empirical analyses, with performance gains reported on both benchmarks.

  • Experimental Setup: METAL is evaluated on Epic-Kitchens-55 and Daily-DA using top-1 target accuracy under the federated domain adaptation setting.The study also includes ablation studies, hyper-parameter sensitivity studies, and empirical analysis.
  • Sensitivity Analysis: The sensitivity analysis varies the number of temporal segments T on Epic-Kitchens-55, where the caption states that performance scales significantly with higher temporal resolution.Frame features are 2048-dimensional and are extracted using a frozen I3D feature extractor.
  • Datasets: Epic-Kitchens-55 contains 55 hours of egocentric kitchen recordings from 32 individuals in 4 cities, producing domain shifts across participants.Three domains with eight shared action classes form three cross-domain tasks, with two source domains and one target domain.
  • Datasets: Daily-DA combines ARID, HMDB51, and Moments-In-Time with eight shared action classes, including challenging adverse illumination conditions from ARID.The experiments use the datasets' specified train-test splits.

4.2. Overall Results and Comparisons

METAL is evaluated against current FDA methods on Epic-Kitchens-55 and Daily-DA using top-1 target accuracy. It outperforms the baselines broadly, although Co-MDA remains stronger for the MIT target domain.

  • The comparisons report top-1 target accuracies for METAL and FDA approaches including KD3A, FDAC, FedGP, and Co-MDA.
  • 18.87% average relative performance gain over current FDA methods is achieved by METAL on Epic-Kitchens-55.
  • 8.38% average performance gain is achieved by METAL on Daily-DA, outperforming all FDA baselines except Co-MDA with MIT as the target domain.
  • The ablation results are averaged across all three Epic-Kitchens-55 tasks.

4.3. Ablation and Hyper-Parameter Sensitivity Study

Ablations show that distinct temporal scales, late fusion, and L2-based scale coordination each contribute to METAL’s performance. Hyper-parameter studies further indicate robustness to scale spacing and several L2 loss weights.

  • Individual scale contribution: Using all three temporal scales achieves higher fusion target accuracy than any subset because each scale contributes distinct temporal information.
  • Fusion method: Late fusion significantly outperforms early fusion because per-scale temporal alignment requires distinct representations at different frequencies.
  • L2 loss: Removing the L2 loss causes a 1.203% performance loss, supporting scale coordination to prevent one scale from dominating.
  • Hyper-parameter sensitivity: METAL achieves similar performance across scale-spacing configurations, including configurations containing a relatively poor coarse scale.
  • Hyper-parameter sensitivity: The chosen L2 loss weight of 0.1 performs best, while 0.05 and 0.2 produce an identical performance plateau.

5. Conclusion

The paper introduces METAL for privacy-preserving federated video domain adaptation, combining multi-scale temporal representations, per-scale distillation, late fusion, and scale coordination. Experiments and ablations across two video adaptation datasets validate this approach.

  • METAL addresses FVDA by learning multi-scale temporal representations and performing per-scale knowledge distillation using only model parameters.
  • Late fusion preserves distinct temporal representations so each scale contributes unique information to the fusion head through feature-based knowledge distillation.
  • Scale coordination prevents any temporal scale from dominating training.
  • Experiments, ablations, and hyper-parameter studies across Epic-Kitchens-55 and Daily-DA validate METAL’s approach to FVDA.
Loading 2608.29186v1…