Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

• Cross-view temporal correspondence must preserve object identity across viewpoints and time despite appearance changes, disappearance, and ambiguous observations.Source: §1 · pp. 1–2
• Track-then-match and match-then-track decouple cross-view matching from temporal tracking, leaving target-view errors or initial-match errors able to propagate.Source: §1 · p. 2, Figure 1 · p. 2
• XTrack unifies cross-view matching and temporal tracking in one prediction process, allowing evidence from one view and full temporal context to resolve uncertainty in the other.Source: §1 · p. 2
• XTrack repurposes pretrained video DiT priors for motion and appearance variation to predict masks jointly from paired ego-exo videos.Source: §1 · p. 2
• Grounding-aware designs combine cross-view attention alignment, hard negative conditioning, and local context crops to strengthen query-specific predictions.Source: §1 · p. 3
• A curation pipeline converts sparse Ego-Exo4D masks into dense supervision using bidirectional propagation, cross-view verification, and within-view semantic filtering.Source: §1 · p. 3
• XTrack achieves competitive performance with robust temporal consistency and identity preservation; ablations support the importance of its flow prediction, grounding, crops, and pretrained video priors.Source: §1 · p. 3

• Cross-view correspondence methods include view-invariant feature matching, mask-level correspondence, and foundation-model adaptation, while many still use frame-level matching or sequential updates.Source: §2 · p. 3
• Video object segmentation preserves identity over time through methods such as memory propagation and query-based prediction, but operates on a single video rather than linking identities across views.Source: §2 · p. 3
• Pretrained video diffusion models learn temporal continuity, object persistence, and appearance-variation priors used in perception, while prior work mostly targets single-view or within-view tasks.Source: §2 · p. 3

• Flow matching transports Gaussian noise toward an encoded input in latent space by learning a field between latent endpoints.Source: §3 · p. 3
• Figure 2 presents the overall architecture of XTrack.Source: Figure 2 · p. 4
• A video DiT computes full self-attention over its entire unified input-token sequence using query, key, and value projections.Source: §3 · p. 4

4.1 Problem Formulation and Overview

• XTrack takes paired ego–exo videos and a first-frame query mask, then jointly predicts mask sequences in both views over time.The query may come from either view, yielding Ego2Exo or Exo2Ego.Source: §4.1 · p. 4

4.2 Cross-view RGB-to-Mask Flow via Video DiT Repurposing

• Flow matching transports paired-video latents from RGB to mask, redirecting a pretrained video DiT toward query-conditioned mask prediction.The RGB latent already encodes scene layout, appearance, and temporal context; the transport direction is zmask − zrgb.Source: §4.2 · pp. 4–5

4.3 Unified Ego-Exo Token Space

• A shared ego–exo token space lets full attention use cross-view evidence at every layer while jointly modeling matching and within-view temporal propagation.The videos are spatially concatenated as a diptych, and the query mask conditions the query side while the opposite side receives an empty mask.Source: §4.3 · p. 5

4.4 Enhancing Cross-View Object Grounding

• Focal attention alignment supervises target-side foreground tokens to attend to query-side foreground, focusing the loss on weakly aligned cases.The alignment loss is small when attention already reaches the correct foreground and applies in either query direction.Source: §4.4 · pp. 5–6
• Hard negative conditioning contrasts predictions from correct and mismatched query masks, discouraging query-agnostic predictions when similar objects are present.The margin loss penalizes similar target-side predictions under different query conditions.Source: §4.4 · p. 6
• Object-centered local crops add fine-grained evidence for small objects while retaining their original spatial positions through RoPE.The crop tokens are concatenated with global ego–exo tokens and supplied alongside the query mask.Source: §4.4 · p. 6

4.5 Training

• Training first uses 2D cross-view cycle consistency, then video-level fine-tuning with dense pseudo-labels to extend matching priors to temporal prediction.The cycle maps a predicted target-view mask back to the query view; the curation pipeline uses bidirectional propagation, cross-view geometric verification, and within-view semantic verification.Source: §4.5 · p. 6, Figure 4 · p. 7

Bidirectional Mask Propagation and Refinement Region Selection

• Bidirectional propagation from annotated anchor frames uses forward–backward mask disagreement to flag chunks for refinement.The pipeline measures mIoU between the two propagated masks and targets low-agreement chunks.Source: §5 · p. 7

Cross-view Geometric Verification

• For unreliable frames, cross-view geometric verification selects an ego-mask candidate whose VGGT feature similarity to the exo reference exceeds a threshold.The exo reference provides same-time evidence; VGGT features are used instead of explicit triangulation, which is unreliable with ego-camera fisheye distortion.Source: §5 · p. 7

Within-view Semantic Verification

• Within-view semantic verification retains a selected candidate only when its DINOv3 feature similarity to a nearby ground-truth anchor exceeds a threshold.The final dense supervision keeps masks meeting both geometric and semantic criteria, producing cleaner, more temporally consistent tracks.Source: §5 · p. 8

6.1 Experimental Setup

• The benchmark uses Ego-Exo4D validation samples with curated dense annotations and human verification, defined by ego-view, exo-view, and object tuples.The evaluation compares three official baselines and four models under Track-then-Match and Match-then-Track settings.Source: §6.1 · p. 8

6.2 Main Results

• Track-then-Match has low VA and mR@IoU because target masks are predicted independently without temporal context, whereas Match-then-Track uses SAM2 to propagate both views.The paper reports that establishing an initial match and then tracking it with SAM2 generally provides greater overall stability than Track-then-Match.Source: §6.2 · pp. 8–9
• XTrack remains highly competitive in visibility accuracy and mR@IoU against a strong SAM2-assisted Match-then-Track baseline, with a trade-off in per-frame J&F precision.The authors attribute the contour-precision trade-off to fine boundaries for small objects being diluted in the compressed video-DiT latent space.Source: §6.2 · p. 9, Table 1 · p. 9

6.3 Ablation Studies

• Adding attention alignment improves query- and target-side prediction; hard negatives improve mask quality, local crops improve overall performance, and removing pretrained weights causes a clear drop.The authors interpret these ablations as evidence for query-specific instance discrimination, fine-grained object evidence, and pretrained video-DiT priors.Source: §6.3 · p. 9

7 Conclusion

• XTrack repurposes a pretrained video DiT for joint view-time mask prediction, using its spatiotemporal and semantic priors to produce query-conditioned masks across views.The paper also introduces a pseudo-label curation pipeline to convert sparse annotations into dense supervision for training.Source: §7 · p. 9

A Challenges of Cross-view Object Correspondence

• Frame-level correspondence can produce empty or incorrect matches when the query is absent or ambiguous, resulting in temporally incoherent target predictions.The paper reformulates the task to use temporal context beyond individual frames to alleviate these ambiguities.Source: A Challenges of Cross-view Object… · p. 16

B.1.1 Unified Ego-Exo Token Space

• The pretrained video DiT processes paired ego and exo clips as a diptych, concatenating their latent grids into one shared spatiotemporal token sequence.The first-frame query mask is placed in the query-side region; its conditioning tokens align with the corresponding visual tokens.Source: §B.1.1 · p. 17

B.1.2 RGB-to-Mask Flow

• XTrack transports paired RGB latents to a full diptych mask latent, decoding temporally coherent mask sequences for both views while retaining the DiT’s priors.Joint processing lets the model use cross-view evidence and temporal context rather than independent per-frame or per-view inference.Source: §B.1.2 · p. 17

B.2.1 Focal Attention Alignment

• Focal attention alignment supervises target-to-query attention using foreground masks, biasing existing cross-view attention blocks without adding a separate matching module.The same formulation applies in both Ego2Exo and Exo2Ego directions.Source: §B.2.1 · pp. 17–18

B.2.2 Hard Negative Conditioning

• Hard negative conditioning exposes the model to another valid object instance and penalizes predictions under a negative query when they resemble positive-query predictions.This is designed to address instance ambiguity between competing objects in the same scene.Source: §B.2.2 · p. 18

B.2.3 Local Context Crop

• For small-object chunks, XTrack appends object-centered crop latents to global diptych tokens, preserving crop positions with RoPE coordinates from the original frame.The crop is randomly dropped during training so the model does not rely exclusively on local context.Source: §B.2.3 · p. 18

B.3.1 Conditioning Mask Augmentation.

• Training varies the first-frame conditioning mask among original, coarse, and bounding-box prompts, all encoded into the same query-side region.Source: §B.3.1 · p. 18

B.4 Implementation Details

• XTrack uses 29-frame samples encoded as eight latent frames, with 448 × 448 ego and 448 × 800 exo scenes, and is trained with AdamW at learning rate 1e−5.Training uses batch size 16 on four H100 GPUs with mixed precision; inference uses 50 diffusion steps and classifier-free guidance scale 3.5.Source: §B.4 · p. 18

C.1 Bidirectional Propagation Details

• SAM2 propagates masks forward from earlier anchors and backward from later anchors, producing two candidate sequences for each annotated temporal chunk.The authors note that occlusion, hand-object interaction, and rapid viewpoint changes can cause propagation errors in ego-view videos.Source: §C.1 · p. 19

C.2 Unreliable Chunk Selection Details

• The pipeline averages per-frame mIoU between forward and backward masks to score chunk reliability, flagging low-scoring chunks and consecutive low-mIoU segments for refinement.Unlike endpoint reconstruction, this check directly measures disagreement at intermediate frames and can detect local failures.Source: §C.2 · p. 19

C.3 Cross-view Geometric Verification Details

• For unreliable chunks with an exo-visible object, VGGT feature similarity and spatial consistency select between forward and backward ego candidates using the exo mask as reference.When the object is absent or fully occluded in the exo view, the pipeline skips this cross-view check and falls back to within-view semantic verification.Source: §C.3 · p. 19

C.4 Within-view Semantic Verification Details

• Within-view verification compares candidate regions with nearby ego anchor masks using DINOv3 cosine similarity, accepting candidates above a threshold and discarding segments when none pass.When cross-view verification is skipped, both directional candidates are checked and the passing candidate with higher similarity is selected.Source: §C.4 · pp. 19–20

C.5 Dataset Examples and Statistics

• The curated dataset contains 708 training videos and 191 validation videos, with 14M and 172K annotated frames, respectively.The training set has 3K tracks and the validation set 300; validation tracks received track-level human screening, with 10% of accepted-track frames sampled for manual inspection.Source: §C.5 · p. 20

D.1 Evaluation Dataset

• Validation comprises 191 takes and 300 tracks with 344K frames across both directions, sampled to preserve task-category and object-scale distributions.Tracks are capped at 600 frames, with longer tracks uniformly subsampled; validation received track-level screening rather than exhaustive frame-by-frame correction.Source: §D.1 · p. 20

D.2 Evaluation Metric

• Performance is assessed along per-frame segmentation quality and temporal consistency, reflecting the task’s requirements for accurate masks and identity preservation.Per-frame measures capture mask quality; Recall@IoU measures whether a correct initialization remains correct over the sequence.Source: §D.2 · p. 20
• J, F, and their average J&F measure region overlap, contour alignment, and balanced region-and-boundary quality, respectively.J&F = 1/2(J + F).Source: §D.2 · p. 20
• Location Error measures object-center distance, while Visibility Accuracy measures whether the model predicts object visibility correctly.These metrics capture localization when overlap is low and visibility changes caused by occlusion or leaving and re-entering the field of view.Source: §D.2 · p. 20
• Recall@IoU tests whether sequences initialized above threshold p remain above it at every later frame, and mR@IoU averages recall over thresholds 0.1, 0.3, 0.5, and 0.7.It measures temporal retention conditional on correct initialization, not initial matching accuracy; the metric is undefined and reported as NaN when no sequence meets the initialization condition.Source: §D.2 · p. 21

E.1 Analysis for Design Choice

• RGB-to-mask transport yields cleaner early representations and more precise masks than noise-to-mask, while reducing attention cost to nearly one quarter when RGB tokens dominate.Noise-to-mask requires approximately 2N + Ncond tokens versus N + Ncond for RGB-to-mask; the comparison reflects quadratic self-attention cost.Source: §E.1 · pp. 21–23

E.2.1 Unified Token Space and the Implicit Matching Problem

• The unified token space permits cross-view and cross-time interaction, but unrestricted self-attention does not ensure that target tokens attend to the correct query object.With visually or semantically similar distractors, implicit matching can disperse attention and yield incorrect or incomplete masks.Source: §E.2.1 · pp. 23–24, §E.2.2 · p. 24

E.2.2 Attention Dispersion as a Root Cause of Failure

• Without alignment, cross-view attention is diffuse and predictions can be mislocalized, fragmented, or absent; focal alignment concentrates attention on the correct object region.The attention maps visualize both directions of cross-view attention alongside the resulting ego-side prediction mask.Source: Figure 10 · p. 23, §E.2.2 · p. 24

E.2.3 Why Focal Attention Alignment Works

• Focal attention alignment directly regularizes bidirectional cross-view attention toward the ground-truth object region, providing a localized signal beyond the mask-prediction loss.The analysis reports improved correspondence accuracy, particularly in complex scenes.Source: §E.2.3 · p. 24

E.3 Effect of Local Context Crops on Small Objects

• On the bottom 25% smallest-object subset, local context crops substantially improve J&F in both directions, with a particularly large relative gain in Ego2Exo.Higher-resolution object-centric evidence supplements global context, but crops alleviate rather than fully resolve boundary errors from latent compression.Source: §E.3 · p. 24, Table 3 · p. 24

F.1 Additional Qualitative Results

• Compared qualitatively with V2-SAM, XTrack more reliably localizes target presence across frames under cross-view appearance changes, although its masks can have less precise boundaries.V2-SAM’s match-then-track pipeline depends heavily on the initial cross-view match, which tracking has limited ability to refine when incorrect or incomplete.Source: §F.1 · pp. 25–26, Figure 11 · p. 25

F.2 Additional Quantitative Results

• XTrack shows strong temporal retention across both directions and particularly competitive performance under challenging Ego2Exo query conditions.Table 4 provides per-threshold Recall@p results for Ego2Exo and Exo2Ego; egocentric queries are more ambiguous because of truncation, camera motion, and occlusion.Source: §F.2 · p. 26, Table 4 · p. 27

F.3 Challenge-based Evaluation

• XTrack achieves higher J&F and VA, and lower LE than the compared baselines across delayed-target-appearance and temporally disjoint-visibility scenarios in both directions.The evaluation uses four subsets of approximately 10K frames each, covering both directions and about 10% of the validation set.Source: §F.3 · pp. 27–28

G.1 Limitations

• Objects occupying only a few latent tokens can lose fine boundaries or appearance cues despite the local context branch.The limitation mainly affects extremely small or heavily compressed objects; local crops improve robustness in most small-object cases.Source: §G.1 · p. 28
• Chunked processing supports long-video inference, but a single forward pass cannot attend to arbitrarily distant frames.The paper suggests extending temporal context or adding memory mechanisms to improve long-range consistency.Source: §G.1 · p. 28
• Iterative denoising makes XTrack slower at inference than feed-forward segmentation or tracking models.The paper identifies faster samplers, distillation, and fewer denoising steps as possible efficiency improvements.Source: §G.1 · p. 28

G.2 Discussion and Future Work

• Future directions include preserving more small-object detail, extending temporal context across chunks, and reducing iterative-denoising cost.Suggested approaches include multi-scale latents or hierarchical tokens, memory or recurrent mechanisms, and fewer-step sampling or distillation.Source: §G.2 · p. 28

H Additional Related Works

• Unlike several prior dataset pipelines, XTrack’s curation starts from existing Ego-Exo4D annotations and uses forward-backward disagreement for per-frame reliability without extra manual labels.Cross-view geometric verification and within-view semantic filtering further refine propagated masks; Figure 15 shows synchronized views with dense annotations, and Figure 16 illustrates the verification stages.Source: H Additional Related Works · pp. 28, 30, Figure 15 · p. 30, Figure 16 · p. 31
• The method repurposes Wan’s pretrained spatiotemporal priors to predict object masks jointly across views and time.The paper situates video DiTs as models of global spatiotemporal interactions and describes Flow Matching as learning a continuous velocity field along a deterministic ODE trajectory.Source: H Additional Related Works · pp. 30–31