PaperlayerView exact revision on arXiv

Source-linked AI summary

Moving Beyond More Views: Redundancy-Aware Ego-Exo Fusion for Proficiency Estimation

Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, Andrew Gilbert

arXiv:2608.25736v1cs.CV

TL;DR

EgoExo proficiency estimation must combine complementary first- and third-person cues without letting redundant views or feature overfitting dilute useful information. The paper introduces AdaMVS for adaptive view selection and VIB-GB for regularised fusion, achieving state-of-the-art results on EgoExo-4D and EgoExo-Fitness. The authors note annotation subjectivity and limited dataset scale as scope limitations.

  • Problem

    Conventional EgoExo fusion can degrade as more views are added because redundant information and feature overfitting hinder robust multiview proficiency estimation.

  • Method

    AdaMVS adaptively selects informative exocentric view tokens, while VIB-GB combines gradient blending with variational information bottleneck regularisation to control fusion overfitting.

  • Results

    State-of-the-art proficiency estimation results were achieved on EgoExo-4D and EgoExo-Fitness, with improved robustness and efficiency.

  • Takeaways & Limitations

    The complementary modules jointly learn which heterogeneous or homogeneous views to fuse and how to regularise their fusion.

  • Takeaways & Limitations

    The framework is evaluated under subjective proficiency annotations and limited dataset scale, motivating extension to broader multiview tasks.

Abstract

from arXiv · show

EgoExo proficiency estimation aims to assess action quality by integrating fine-grained motion cues from egocentric (1st-person) views with spatial context from multiple exocentric (3rd-person) views. Simply adding more exocentric views degrades EgoExo performance, as redundant or noisy perspectives dilute useful motion cues. Our analysis identifies two key causes: (1) Multiview redundancy - From the data perspective, certain views provide limited or noisy information, diluting discriminative cues; (2) Overfitting - From the feature perspective, conventional fusion increases representational complexity, causing the model to memorise view-specific patterns rather than learn generalisable representations. To address these issues, we propose two complementary modules: AdaMVS, which adaptively identifies and fuses the most informative view tokens under weak supervision from the data perspective, and VIB-GB, which combines Gradient Blending and Variational Information Bottleneck regularisation from the feature perspective to compress redundant signals and suppress overfitting during training. Experiments on EgoExo-4D and EgoExo-Fitness demonstrate that our method learns both which view to look at and how to fuse them, achieving new state-of-the-art results. Our source code is available at https://github.com/dx199771/AdaMVS

1 Introduction

EgoExo proficiency estimation combines fine-grained egocentric motion cues with exocentric spatial context, but redundant views and overfitting make conventional fusion unreliable. AdaMVS adaptively selects informative views, while VIB-GB regularises fusion to improve robustness and efficiency.

  • EgoExo estimation combines subtle egocentric motion and intention cues with exocentric full-body dynamics and spatial context.
  • Adding exocentric views can degrade performance because heterogeneous, unaligned views may be noisy, redundant, occluded, blurred, or irrelevant.
  • AdaMVS adaptively scores and selects the most informative Top-K exocentric views before token-level fusion, reducing redundancy and computational cost.
  • VIB-GB complements view selection by controlling overfitting during feature fusion, jointly learning what to fuse and how to regularise fusion.
  • Experiments on EgoExo-4D and EgoExo-Fitness achieve state-of-the-art proficiency estimation with improved robustness and efficiency.

2 Related Work

Prior EgoExo work combines complementary viewpoints but remains limited by difficult cross-view alignment, redundancy, noise, and overfitting. AdaMVS with VIB-GB addresses these issues through adaptive token-level selection and regularised fusion.

  • Egocentric and Exocentric Understanding: Egocentric views capture hand–object interactions and attention cues, whereas exocentric views provide holistic body motion and scene context.
  • Egocentric and Exocentric Understanding: Existing cross-view approaches use knowledge transfer, view-invariant features, or temporal alignment, but limited paired data constrains scalability and invariance does not remove redundant noise.
  • Proficiency Estimation: Proficiency estimation differs from AQA by classifying discrete skill levels rather than predicting continuous quality scores, while existing methods primarily rely on late fusion.
  • Multi-view Learning: Multi-view fusion methods face costly processing, weak adaptation to changing view informativeness, and overfitting risks caused by differing perspectives and generalisation rates.
  • Multi-view Learning: AdaMVS and VIB-GB provide a unified solution intended to jointly address feature redundancy and overfitting in heterogeneous Ego–Exo data.

3 Methodology

The framework addresses multiview proficiency estimation by selecting informative exocentric views and regularising ego–exo fusion to reduce redundancy and overfitting.

  • AdaMVS: Exo-AdaMVS selects the Top-K most informative exocentric views before fusion to mitigate redundancy.View scores are produced from token-level clip features and aggregated into view-level importance scores.
  • AdaMVS: AdaMVS uses token-level importance scores to focus on discriminative views while suppressing redundant ones.Dominant views capture 83% of cumulative softmax attention weights during fusion.
  • AdaMVS: Gumbel-Softmax weighting is used during training, while Soft Fusion at inference retains complementary cues while filtering noise.The selected views are aggregated with normalized importance scores before exocentric decoding.
  • VIB-GB: VIB-GB combines OGR-guided gradient reweighting with Variational Information Bottleneck regularisation across ego and exo branches.Gradient weights are updated online to down-scale branches that overfit faster, while VIB compresses branch latent spaces.
  • VIB-GB: The combined framework yields compact and generalisable proficiency representations by reducing redundant signals and mitigating fusion overfitting.The method jointly addresses redundancy at the data level and overfitting at the feature level.

4 Experiments

Experiments on Ego-Exo4D and EgoExo-Fitness evaluate accuracy, efficiency, view selection, and overfitting mitigation. AdaMVS and VIB-GB consistently improve fusion robustness, with complementary gains from adaptive selection and feature regularisation.

  • Comparison with State-of-the-Art: AdaMVS achieves a 5.5% absolute accuracy improvement over SkillFormer on Ego-Exo4D and a 7.1% absolute improvement on EgoExo-Fitness.Using identical feature extraction settings, gains across K400, K600, and HowTo100M are 12.7%, 3.1%, and 9.1%, respectively.
  • Fusion and Efficiency: AdaMVS-Small reaches similar accuracy with 0.26 GFLOPs and 2.19 M parameters, reducing computation and model size by over 11×.The lightweight projection layer reduces complexity without degrading representation quality.
  • Ablation Study: Starting from 45.5%, the ego–exo ablation reaches 48.0% with AdaMVS, 48.6% with Gradient Blending, and 51.0% when VIB is added.AdaMVS filters redundant or noisy views, while VIB regularises the latent space and filters non-essential information.
  • Module Complementarity: VIB-GB applied to the naive baseline yields 48.7% accuracy, while the full model reaches 51.0%, demonstrating complementary contributions from data- and feature-level interventions.AdaMVS suppresses uninformative views; VIB-GB limits memorisation of view-specific noise.
  • View Selection: AdaMVS performs best with K = 2 exocentric views, consistent with two dominant views accounting for approximately 82.8% of videos.Additional views mainly introduce redundant or noisy information.
  • Overfitting Analysis: With VIB-GB, accuracy reaches 53.0% and OGR stabilises after around 20 epochs, whereas the AdaMVS baseline begins overfitting around epoch 10 and reaches 48.7%.VIB-GB produces smoother training, a weaker overfitting trend, and declining OGR after an initial rise.

5 Conclusion

The framework addresses redundant information and overfitting in Ego–Exo fusion through complementary data- and feature-level modules, with reported robustness and broader multiview applicability.

  • AdaMVS adaptively selects informative exocentric views, while VIB-GB combines a variational information bottleneck with gradient blending to alleviate overfitting.
  • The two modules are reported to be highly complementary and jointly enhance the overall robustness of the system.
  • The modular framework is presented as a general principle for adaptive fusion across heterogeneous ego-exo and homogeneous exos camera views.
  • Subjective proficiency annotations and limited dataset scale remain stated challenges and scope boundaries.

Moving Beyond More Views: Redundancy-Aware Ego–Exo Fusion for Proficiency Estimation – Supplementary File

The supplementary file lists the paper’s authors: Xu Dong, Wanqing Li, Anthony Adeyemi-Ejeye, and Andrew Gilbert.

  • Xu Dong is listed as an author.
  • Wanqing Li is listed as an author.
  • Anthony Adeyemi-Ejeye and Andrew Gilbert are also listed as authors.

A Additional Results

The additional results show AdaMVS selecting informative views and maintaining more stable performance than conventional fusion as exocentric views increase, while selection remains ambiguous when views are highly similar.

  • Additional Visualizations: AdaMVS assigns high weights to informative viewpoints and suppresses views affected by occlusion, limited motion cues, or background noise.
  • Additional Visualizations: When exocentric views have highly similar content or comparable performer visibility, AdaMVS produces relatively uniform importance scores because view diversity is weak.
  • View-wise Fusion Analysis: View-wise fusion results evaluate individual cameras, increasing exocentric-view counts, all-view permutations, and ego–exo fusion configurations.
  • View-wise Fusion Analysis: Conventional fusion can decline with additional views, whereas AdaMVS maintains stable gains by selecting informative and complementary viewpoints for each action instance.
  • Redundancy/Overfitting Analysis: Adding identical views reduced performance, while using all exocentric views underperformed selective fusion; AdaMVS achieved 49.2.
  • Ego + Single-exo-view Comparison: The dominant exocentric view varies across methods, so a single fixed view is insufficient; AdaMVS provides task-agnostic robustness without manual tuning.

B Metrics for Quantifying Multiview Redundancy

The paper characterizes multiview redundancy through low task informativeness and high representational overlap, quantified jointly with MI and CKA.

  • Redundancy Factors: Multiview redundancy arises from views contributing little to proficiency prediction or encoding information already present in other views.
  • Redundancy Metrics: Mutual Information measures each view’s task relevance, while CKA measures representation-level similarity across views.
  • Redundancy Metrics: Together, MI and CKA provide a redundancy measure that evaluates both informativeness and inter-view overlap.
  • Mutual Information: In the paper’s setting, each view’s feature embedding is v and the ordinal proficiency score is y.
  • Mutual Information: Higher MI indicates stronger task-relevant proficiency cues, whereas low MI suggests a noisy, weak, or uninformative view.

Centred Kernel Alignment (CKA)

CKA measures representational similarity to diagnose multiview redundancy, while the redundancy index combines mean cross-view similarity with mutual information to rank views. The section also introduces OGR and its simplified overfitting-focused variant for gradient reweighting.

  • Centred Kernel Alignment (CKA): CKA quantifies similarity between feature representations from two views, ranging from 0 for dissimilar representations to 1 for identical representations up to rotation.Its invariance to isotropic scaling and orthogonal transformations supports redundancy diagnosis.
  • Centred Kernel Alignment (CKA): The redundancy index uses mean CKA similarity across a view’s remaining views divided by mutual information plus epsilon.Lower values indicate more informative, less redundant views, whereas higher values indicate duplicative or noisy views.
  • Centred Kernel Alignment (CKA): Views with low redundancy-index values consistently receive higher learned AdaMVS importance weights, indicating alignment between adaptive selection and data redundancy patterns.AdaMVS therefore prioritises informative and complementary views.
  • Centred Kernel Alignment (CKA): OGR measures overfitting growth relative to generalisation improvement, with higher values indicating optimisation dominated by overfitting.Gradients with lower OGR receive larger weights, while overfitting-prone gradients are down-weighted.
  • Centred Kernel Alignment (CKA): Because validation losses across views are highly correlated, the method simplifies OGR to focus solely on overfitting increments for stable view-sensitive gradient reweighting.Overfitting increments separate ego and exo branches more consistently than generalisation gains.

C Methodology

The methodology combines Transformer-based decoding with adaptive token- and view-level selection for heterogeneous Ego–Exo inputs. It complements this data-level selection with feature-compression and OGR-guided regularisation to reduce redundancy and overfitting.

  • C Methodology: The DETR-style decoder refines learnable queries through self-attention and then interacts with encoded feature sequences through cross-attention.Feed-forward networks, residual connections, and layer normalisation support stable optimisation.
  • C Methodology: AdaMVS dynamically evaluates and selects informative views or tokens in separate egocentric and exocentric branches built on a Transformer architecture.Both branches share frozen-backbone feature extraction and learnable queries acting as semantic anchors.
  • C Methodology: The exocentric branch scores tokens, applies Gumbel–Softmax weighting, averages token weights into per-view scores, selects Top-K views during training, and fuses them after renormalisation.The algorithm explicitly targets discriminative external camera views while pruning redundant perspectives.
  • C Methodology: The framework addresses heterogeneous, asynchronous Ego–Exo data by introducing view adaptivity and token-wise interaction beyond fixed-level fusion strategies.Prior fixed early, intermediate, or late fusion methods struggle to adapt feature relevance over time.
  • C Methodology: VIB-GB processes parallel ego and exo branches through variational information bottlenecks and combines their overfitting increments with OGR-based gradient blending.This couples information compression with dynamic training-dynamics regularisation.

E Limitations and Future Work

The framework retains limitations when views contribute nearly equivalent information, because adaptive selection cannot fully resolve indistinguishable perspectives. The supplied passages provide no future-work detail for this section.

  • E Limitations and Future Work: Adaptive Top-K selection mitigates but cannot fully resolve cases where multiple views contribute similarly.The limitation concerns ambiguity among views rather than a stated failure of the overall framework.

3. Adaptive weighting

Adaptive modality weighting fuses projected ego and exo representations using learned soft weights before downstream decoding. Its practical boundary appears when overlapping views carry nearly equivalent information, while future work proposes dynamically varying the number of selected views.

  • 3. Adaptive weighting: The fusion stage projects both modalities, computes modality weights with Gumbel–Softmax, broadcasts them across tokens, and produces fused tokens for downstream decoding.The procedure supports soft fusion across ego and exo streams.
  • 3. Adaptive weighting: When camera coverage overlaps heavily or motions are simple, AdaMVS may assign nearly uniform weights because views provide almost equivalent information.In these cases it cannot reliably identify a clearly dominant view.
  • 3. Adaptive weighting: Future work will explore adaptive K-selection so the number of selected views varies with action complexity and cross-view variability.The stated goal is to improve dynamic view-count selection.
Loading 2608.25736v1…