PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Cross-View Object Correspondence via Cycle-Consistent Mask Prediction

Shannan Yan, Leqi Zheng, Keyu Lv, Jingchen Ni, Hongyang Wei, Jiajun Zhang, Guangting Wang, Jing Lyu, Chun Yuan, Fengyun Rao

arXiv:2602.18996v2cs.CV

TL;DR

The paper studies object-level correspondence across challenging egocentric and exocentric video viewpoints. It uses conditional binary segmentation with cycle-consistent mask prediction and test-time training, achieving strong benchmark performance on Ego-Exo4D and HANDAL-X. The method remains limited by persistent failures and rare invisible-object cases.

  • Problem

    Cross-view object correspondence is challenging because viewpoint changes cause appearance variation, occlusions, and disjoint spatial references.

  • Method

    The method uses conditional binary segmentation with source-mask conditioning, cycle-consistent round-trip reconstruction, and test-time training.

  • Results

    The method achieves strong performance on Ego-Exo4D and HANDAL-X, including over 3.10% improvement over prior baselines in Exo Query and 36.0% over ObjectRelator on HANDAL-X zero-shot segmentation.

  • Takeaways & Limitations

    Cycle-consistent self-supervision and test-time training improve correspondence under viewpoint shift without requiring complex pipelines or large temporal context.

  • Takeaways & Limitations

    Some errors persist after test-time training, including incomplete regions, visually similar distractors, and failures to detect the target object.

Abstract

from arXiv · show

We study the task of establishing object-level visual correspondence across different viewpoints in videos, focusing on the challenging egocentric-to-exocentric and exocentric-to-egocentric scenarios. We propose a simple yet effective framework based on conditional binary segmentation, where an object query mask is encoded into a latent representation to guide the localization of the corresponding object in a target video. To encourage robust, view-invariant representations, we introduce a cycle-consistency training objective: the predicted mask in the target view is projected back to the source view to reconstruct the original query mask. This bidirectional constraint provides a strong self-supervisory signal without requiring ground-truth annotations and enables test-time training (TTT) at inference. Experiments on the Ego-Exo4D and HANDAL-X benchmarks demonstrate the effectiveness of our optimization objective and TTT strategy, achieving state-of-the-art performance. The code is available at https://github.com/shannany0606/CCMP.

1. Introduction

The paper addresses difficult object correspondence between egocentric and exocentric views with a conditional segmentation framework and cycle-consistent self-supervision. Evaluations on Ego-Exo4D and HANDAL-X show strong performance across challenging viewpoint changes.

  • Motivation: Cross-view correspondence is difficult because egocentric and exocentric cameras create appearance variation, occlusions, and disjoint spatial references.Objects may also differ in lighting, resolution, motion blur, spatial context, and surrounding layout across views.
  • Approach: The framework formulates correspondence as conditional binary segmentation, using source-image information to predict a target-view object mask.A conditioning token injects source information into a DINOv3-based vision transformer with minimal architectural changes.
  • Approach: Cycle consistency transfers the predicted target mask back to the source view to reconstruct the original query mask, providing self-supervision without target-view annotations.The same objective also supports test-time training during inference under domain shift or distributional variation.
  • Results: Over 3.10% improvement over all prior baselines is reported in Ego-Exo4D’s Exo Query setup.In Ego Query, the method reaches 41.95% IoU versus 42.57% for O-MaMa.
  • Results: 36.0% improvement over ObjectRelator is reported on HANDAL-X zero-shot segmentation, with cycle consistency and test-time training improving both setups.The paper reports strong correspondence under extreme viewpoint changes.

2. Related Works

Related work spans cross-view video understanding, vision foundation models, and test-time training. The paper positions its approach within emerging efforts to connect egocentric and exocentric views and adapt models at inference.

  • Cross-view Video Understanding: Most prior video-understanding work targets a single view, while Ego-Exo4D provides time-aligned annotations connecting egocentric and exocentric perspectives.Earlier research covered classification, question answering, and captioning, with cross-view connections remaining limited.
  • Cross-view Video Understanding: Recent cross-view methods include predictive cycle consistency with pseudo-labeling and ObjectRelator’s view-invariant embedding modules.These approaches build on paired masks or auxiliary modules for cross-view correspondence.
  • Vision Foundation Models: Vision foundation models advance representation learning through transformer architectures and self-supervised techniques, providing the broader modeling context for this work.The related work discusses ViT, DeiT, MoCo v3, and MAE.
  • Test-Time Training: Test-time training adapts models using self-supervision on unlabeled test samples, and this paper reports its first successful application to this task with clear improvement.Prior applications span images, videos, language, and generative video transformers.

3. Approach

The approach uses a transformer pipeline to condition target-view segmentation on a masked source object, combining mask, auxiliary, and cycle-consistency objectives. The cycle constraint reconstructs the source mask and supports test-time training without target-mask annotations.

  • Source Feature Extractor: The pipeline extracts source-image features, pools them using the object mask into an object-specific representation, and projects that representation onto a condition token.The source mask is resized and normalized before weighted feature averaging.
  • Transformer Encoder: The target image is patchified and processed with condition and class tokens, allowing conditioned transformer features to guide target-view object representation.The input sequence is [CLS, CDT, x1, x2, . . . , xn].
  • Multi-task Decoder: Two decoder heads generate target-mask features and classify whether the source object is visible in the target image.A lightweight convolutional head produces the final target segmentation mask.
  • Objective Function: The total objective combines mask, auxiliary, and cycle-consistency losses, with hyperparameters balancing the auxiliary and cycle terms.The mask loss combines binary cross-entropy and Dice loss, while auxiliary supervision uses intermediate predictions.
  • Objective Function: Cycle consistency maps the source mask to a predicted target mask and back to reconstruct the original source mask, without requiring ground-truth target masks.This closed-loop objective supports inference-time test-time training; invisible objects are not explicitly handled because they are rare in Ego-Exo4D.
  • Test-Time Training: For test-time training, only the last K transformer encoder layers are fine-tuned for each test pair using the cycle-consistency loss.The procedure uses T gradient update steps with learning rate lr_ttt.

4. Experiments

Experiments on Ego-Exo4D and HANDAL-X evaluate benchmark performance, generalization, ablations, and qualitative behavior across viewpoints, object sizes, and scenarios. The method shows strong results overall, while small target objects remain challenging.

  • Benchmark Results: 44.57% mIoU on Ego-Exo4D represents a +2.9% relative improvement over O-MaMa, with gains up to +13.4% in Contour Accuracy.The method also achieves a +7.0% relative IoU improvement in the Exo Query setup and competitive performance in Ego Query.
  • Benchmark Results: 78.8% IoU on HANDAL-X without training on that dataset surpasses all baselines by a relative improvement of 84.1%.Performance remains superior after fine-tuning on HANDAL-X.
  • Generalization: Exposure to Ego-Exo4D training data improves generalization to unseen HANDAL-X scenes and viewpoints for all evaluated methods.HANDAL-X provides a complementary setting with broader viewpoint variations than the egocentric–exocentric Ego-Exo4D benchmark.
  • Ablation Study: Removing either the cycle-consistency or auxiliary loss degrades performance, while test-time tuning consistently improves results.Removing Lcycle causes a noticeable mIoU drop because it provides the self-supervised signal needed for effective TTT.
  • Ablation Study: Each data-preprocessing strategy contributes to robust and generalizable segmentation, as removing any one degrades per-view IoU and overall mIoU.The strategies include unified bidirectional training, same-view exemplar synthesis, and cross-view preprocessing.
  • Robustness Analysis: Objects occupying more than 0.1% of the target image are handled well, whereas smaller objects remain challenging across viewpoint settings.Differences in object-size distributions help explain why overall Ego2Exo IoU is lower than Exo2Ego IoU despite higher Ego2Exo IoU within each size bin.

5. Conclusion

The paper presents a simple approach for egocentric–exocentric object correspondence that combines training design, cycle-consistent loss, and test-time training. It reports strong performance on Ego-Exo4D and HANDAL-X while addressing viewpoint shifts, occlusion, and motion.

  • The method unifies cross-view correspondence tasks and enriches training with same-view and temporally relaxed pairs.
  • Cycle-consistent loss and test-time training strengthen correspondence under viewpoint variation without requiring complex pipelines or large temporal context.
  • Experiments on Ego-Exo4D and HANDAL-X demonstrate strong performance and robustness to viewpoint shift, occlusion, and motion.

Prediction

This section organizes supplementary material covering training details, test-time training, ablations, efficiency, qualitative results, and limitations. The paper uses TTT as the abbreviation for test-time training.

  • Additional training details are provided in Section B, while test-time training on HANDAL-X appears in Section C.
  • Further ablation studies, efficiency analysis, and qualitative results are provided in Sections D, E, and F.
  • Section G provides limitations and future work, and the paper abbreviates test-time training as TTT.

B. More Training Details

The training details describe a two-stage Ego-Exo4D optimization process, separate HANDAL-X training settings, and an accompanying evaluation table. The Ego-Exo4D process begins with linear probing before all parameters are optimized.

  • Ego-Exo4D training uses AdamW, cosine decay with linear warm-up, batch size 16, and 512 × 512 images.
  • The Ego-Exo4D process first freezes both DINOv3 backbones for 64K linear-probing iterations, then unfreezes all parameters.
  • HANDAL-X training runs for 10 epochs, with offline training taking approximately 2 hours and TTT requiring an additional 1 hour.
  • Table 8 reports evaluation results on the HANDAL-X benchmark.

C. Test-time Training on HANDAL-X

On HANDAL-X, the method already surpasses all baselines without TTT, while Table 8 reports additional TTT results. TTT provides only marginal gains when baseline IoU is already very high.

  • Without TTT, the method already surpasses all baselines on HANDAL-X.
  • Table 8 presents quantitative HANDAL-X results with TTT to demonstrate effectiveness and generalization across benchmarks.
  • When baseline IoU is already very high, TTT yields only marginal improvements.

D. More Ablation Study

The ablations examine mask-prediction design, Dice-loss weighting, test-time optimization steps, layer adaptation, and the performance–latency trade-off. Direct mask prediction performs better, while TTT benefits depend on update budget and adaptation depth.

  • Mask Prediction Method: Direct mask prediction outperforms predicting masks conditioned on object features.The comparison is reported in Table 9.
  • Dice weight: The Dice Loss addresses class imbalance when the target occupies a small mask region and encourages closer predicted-to-ground-truth alignment.Its weight is evaluated in the ablation of the mask supervision objective.
  • Gradient Update Steps of TTT: Ego2Exo reaches its best TTT performance with 2 update steps, whereas Exo2Ego peaks at 7 or more steps.The default Exo2Ego setting is 6 steps because it balances efficiency and performance.
  • Fine-tuning Layers of TTT: Ego2Exo performs best when adapting only a small number of layers, while Exo2Ego benefits from updating substantially more layers.Deeper adaptation yields diminishing returns or slight degradation for Ego2Exo.
  • Performance–Latency Trade-off: Most TTT performance gain is achieved with 2 gradient updates, after which further updates provide diminishing returns.The efficiency analysis reports mIoU as a function of inference time while varying optimization steps.

F. More Qualitative Results

Additional qualitative results span Ego-Exo4D and HANDAL-X across varied activities, viewpoints, and hand–object interactions. The examples show accurate target-mask recovery despite scale, perspective, appearance, occlusion, and scene-context variation.

  • Ego-Exo4D: The Ego-Exo4D examples cover cooking, health, bike repair, music, basketball, and soccer across Ego2Exo and Exo2Ego.Cooking occupies three rows because it is the largest benchmark scenario.
  • Efficiency: The qualitative results are presented alongside a performance–latency analysis that varies the number of test-time optimization steps.The analysis reports mIoU as a function of inference time.
  • Ego-Exo4D: Across Ego-Exo4D scenarios, masks closely match ground-truth annotations despite variations in scene context, object category, occlusion, and viewpoint.The qualitative results also illustrate the effectiveness of TTT in focusing the model more accurately on the target.
  • HANDAL-X: HANDAL-X examples span six diverse hand–object interaction categories, and the method recovers target masks in most cases.The figure highlights the effectiveness of TTT.

G. Limitation and Future Work

The reported failure patterns include incomplete ground-truth coverage, attraction to visually similar objects, and failure to detect the target. TTT partially mitigates these errors, while future work targets temporal cues to reduce them further.

  • Limitations: Common failures are incomplete ground-truth coverage, attraction to visually similar objects, and complete failure to detect the target.The failures are summarized from frequent to rare.
  • Limitations: TTT partially mitigates the identified errors, but some failures persist.The authors identify room for further improvement.
  • Future Work: Future work will incorporate temporal cues to better capture object dynamics and reduce the identified failure patterns.
  • Additional Results: Additional qualitative figures provide representative samples for Ego-Exo4D and HANDAL-X, but their captions alone do not state a performance outcome.
Loading 2602.18996v2…