Source-linked AI summary
Beyond Spatio-Temporal Priors: A Generalizable Approach for Dense Correspondence Matching
Luping Liu, Bingyi Kang, Yifan Wang, Dong Xu
TL;DR
Classical spatio-temporal priors are limited when IEG transformations preserve identity but break physical continuity. FreeMatching combines foundation representations, heterogeneous supervision, and teacher-guided refinement; it improves IEG correspondence while remaining competitive on classical benchmarks.
Problem
Smooth-motion and rigid-geometry priors used in classical correspondence do not cover IEG transformations that preserve visual identity while breaking physical continuity.
Source: §1 · p. 1Method
FreeMatching combines foundation-model representations, heterogeneous correspondence supervision, and teacher-guided iterative refinement for identity-preserving matching.
Source: §1 · pp. 1–2Results
A single model substantially improves correspondence quality on IEG pairs while remaining competitive on classical benchmarks, and its identity-preservation scores correlate with human judgment.
Source: §1 · p. 2Takeaways & Limitations
FreeMatching supports spatially resolved assessment of identity preservation in IEG and serves as a quantitative metric for evaluating consistency in novel data.
Source: §1 · p. 2, §5 · p. 10The model can be challenged by severe occlusions, topological changes, and drastic non-rigid deformations; its ceiling depends on foundation models, and IEG-Bench needs broader categories.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Dense correspondence matching has historically been bounded by simplifying spatio-temporal priors, such as smooth motion and rigid geometry. While effective for classical tasks, these assumptions break down in image editing and reference-guided generation (IEG), where transformations can preserve visual identity while breaking physical continuity. To establish identity-preserving correspondence across such transformations, we introduce FreeMatching, a generalizable framework combining generative and semantic foundation representations with heterogeneous supervision from classical datasets, tracked videos, and synthetic scenes. Teacher-guided iterative refinement further improves correspondence in IEG without dense correspondence annotations. Experimentally, a single FreeMatching model substantially improves correspondence quality on challenging IEG image pairs while retaining competitive performance on classical benchmarks. Furthermore, we demonstrate its utility as a quantitative metric for evaluating identity preservation, with scores that correlate with human judgment. The code is available at https://github.com/luping-liu/FreeMatching.
1 Introduction
FreeMatching targets identity-preserving correspondence across transformations that defeat classical motion and geometry priors. It combines foundation representations, heterogeneous supervision, and iterative refinement, improving IEG matching while remaining competitive on classical tasks.
1 Introduction
- IEG transformations can preserve visual identity while breaking physical continuity, challenging methods built around smooth motion and rigid geometry.The paper frames identity-preserving correspondence as matching the same instance despite substantial changes in appearance and configuration.Source: §1 · p. 1
- FreeMatching combines foundation-model representations, heterogeneous correspondence supervision, and teacher-guided iterative refinement to extend matching to IEG.The framework aims to learn identity-preserving correspondence across the tasks illustrated in Figure 1.Source: §1 · pp. 1–2, Figure 1 · p. 2
- Training uses FLUX2-4B and DINOv3 representations, classical, tracked-video, and synthetic data, then teacher-guided refinement on IEG pairs without dense annotations.Huber loss mitigates annotation noise in the supervised stage; iterative refinement improves local correspondences and provides covisibility supervision.Source: §1 · p. 2
- FreeMatching is competitive on classical benchmarks, substantially improves IEG correspondence, and provides an identity-preservation metric whose scores correlate with human judgment.The metric supports spatially resolved assessment of identity preservation.Source: §1 · p. 2
2 Related Work
Prior work developed classical correspondence methods, unified traditional tasks, and explored semantic matching and diverse training paradigms. FreeMatching builds on these directions to extend one model to IEG.
2.1 Classical and Unified Correspondence
2.2 Semantic Correspondence and Foundation Features
- Semantic correspondence work addresses geometry and viewpoint sensitivity [23] [24], representation efficiency [25] [26], and generalization [27], while DUSt3R [28] and VGGT [29] provide geometry-grounded features.Recent work also adapts VGGT priors to dense semantic matching [30].Source: §2.2 · p. 3
2.3 Learning Paradigms for Correspondence
- Training has expanded from synthetic datasets such as FlyingThings [18] toward data distillation [31] and diverse data sources [32]; FreeMatching adds tracked videos, synthetic and classical data, and teacher-guided IEG refinement.This combination supports instance-level matching across appearance, pose, and scene-composition changes while retaining classical capabilities.Source: §2.3 · p. 3
3 FreeMatching
FreeMatching treats dense correspondence as identity-preserving matching beyond rigid geometry and smooth motion, and combines foundation representations, heterogeneous supervision, iterative refinement, and reconstruction-based evaluation.
3 FreeMatching
- FreeMatching extends classical correspondence matching to image editing and reference-guided generation, with reconstruction-based evaluation of matching quality and identity-preserving consistency.Source: §3 · p. 3
3.1 Task Definition
- Correspondence matches regions depicting the same instance across changes in viewpoint, pose, appearance, and scene composition, wherever an identifiable counterpart remains.Source: §3.1 · p. 3
3.2 Foundation Models
- The model combines FLUX generative representations with DINOv3 semantic features to predict correspondence and covisibility in one forward pass.Source: §3.2 · pp. 3–4, Figure 2 · p. 4
3.3 Supervised Pre-training
- Supervised pre-training combines classical datasets with 520k tracked-video clips and 360k synthetic image pairs.Source: §3.3 · p. 4, Table 1 · p. 4
- Huber correspondence regression and L2 covisibility-mask supervision make the objective robust to noisy real-world annotations while using clean synthetic data for precise matching.Source: §3.3 · pp. 4–5
3.4 Teacher-Guided Iterative Refinement
- An EMA student supplies initial alignments, and a fixed teacher estimates local corrections that are composed into pseudo-targets for student training without dense IEG annotations.Source: §3.4 · p. 5
3.5 Consistency Evaluation
- One FreeMatching model is reported competitive across outdoor and indoor geometric matching and downstream pose estimation.Source: Table 2 · p. 6
- Reconstruction metrics assess correspondence on consistent benchmarks and, for new image pairs, can indicate content inconsistency within covisible regions, subject to alignment accuracy.Source: §3.5 · pp. 5–6
4 Experiments
FreeMatching achieves the strongest reconstruction scores among compared methods on IEG-Bench, including after weak adaptation, while maintaining competitive performance on classical benchmarks. Experiments also examine qualitative alignment, local identity assessment, and staged training contributions.
4.1 Classical Benchmarks
- On classical benchmarks, FreeMatching beats similarly adapted UFM and RoMa on ScanNet and ETH3D, but UniMatch has the lowest optical-flow EPE among methods shown.Adapted UFM is stronger on Sintel Clean and KITTI, while FreeMatching is stronger on Sintel Final.Source: §4.1 · p. 6, Table 3 · p. 7
4.2 Identity-Preserving Correspondence
- FreeMatching achieves the strongest reconstruction scores among compared methods on IEG-Bench.Table 4 reports SigLIP 2 and DINOv3 similarity, Ref-MSE, and LPIPS reconstruction measures.Source: §4.2 · p. 7, Table 4 · p. 7
- After 10k adaptation updates, FreeMatching achieves the best scores across all four Table 5 metrics, consistent with advantages across metrics and evaluation settings.The comparison uses UNO+OmniGen2 refinement data and includes reference-mask and full-image evaluations.Source: §4.2 · p. 7, Table 5 · p. 8
4.3 Qualitative Results
- In qualitative IEG examples, substantial changes in appearance, pose, and composition produce distortions or incomplete matches for compared methods, while FreeMatching aligns more coherently.Figure 4 visualizes warping Image A toward Image B, with references from B composed into A.Source: §4.3 · p. 8, Figure 4 · p. 8
4.4 Fine-Grained Identity Assessment
- Local heatmaps reveal texture or structure deviations that global metrics can mask, including inconsistent balloon text after alignment.The assessment computes perceptual divergence within covisible regions.Source: §4.4 · p. 8, Figure 5 · p. 9
4.5 Stage-wise System Construction
- Cumulative construction attributes IEG-Bench matching gains to FLUX initialization, DINOv3 features, Video/Blender supervision, and teacher-guided refinement.Table 6 reports DINOv3 reconstruction similarity as components are added.Source: §4.5 · p. 9, Table 6 · p. 9
- Both flow-domain and IEG-domain refinement improve IEG-Bench Ref-MSE over Stage 1, with gains varying by domain; 10k-IEG lowers EPE on all five classical benchmarks.10k-Flow performs best on Sintel Clean, while 10k-IEG performs best on the remaining reported benchmarks.Source: §4.5 · p. 9
5 Discussion
FreeMatching extends dense correspondence to identity-preserving matches across IEG transformations while retaining competitive classical performance. The discussion distinguishes this goal from semantic matching and notes boundaries in difficult transformations, foundation-model capability, and benchmark coverage.
5 Discussion
- A single model establishes identity-preserving correspondences across diverse visual transformations while retaining competitive classical benchmark performance.The framework combines foundation-model representations, heterogeneous supervision, and teacher-guided iterative refinement.Source: §5 · pp. 9–10
- Generative ambiguity such as object replication is not modeled by the paper’s efficient single-pass regression experiments; a probabilistic diffusion extension is presented as a route to multiple plausible matches.Source: §5 · p. 10
- FreeMatching targets the same instance and its local details across IEG pairs, rather than semantically related regions that may belong to different instances.The displayed multi-object comparison illustrates more consistent preservation of object shape and appearance.Source: §5 · p. 10
- Extreme occlusions, topological changes, and drastic non-rigid deformations can challenge the model; its ceiling depends on foundation models, and IEG-Bench needs broader category coverage.Source: §5 · p. 10
A Architecture
The architecture combines a generative DiT prior and DINOv3 semantic features to predict dense correspondence and covisibility. It processes image pairs jointly and produces both outputs in one forward pass.
A Architecture
- The framework uses a Diffusion Transformer’s generative prior and DINOv3 semantic robustness to predict dense correspondence and covisibility masks.Source: A Architecture · p. 15
A.1 Architecture Overview
- The backbone initializes from FLUX.2-klein-base-4B, retains its latent-token input projection, and adds semantic feature injection and a correspondence decoder.A frozen VAE encodes images into packed latent tokens, while a frozen DINOv3 encoder supplies projected patch features.Source: §A.1 · p. 15
A.2 Input Representation and Conditioning
- The source and target token sequences are concatenated, with position encodings retaining spatial coordinates and image identity for joint attention.Correspondence prediction uses cached empty-prompt embeddings and requires only the image pair.Source: §A.2 · p. 15
A.3 Prediction and Output Decoding
- A task-specific decoder produces full-resolution correspondence coordinates and a covisibility mask in a single forward pass.Source: §A.3 · p. 15
- Inference latency and peak allocated memory are measured at a common external input resolution on an H20 GPU; improving inference efficiency remains future work.Source: §A.3 · p. 15
B Datasets
- The dataset strategy combines established benchmarks with novel large-scale datasets for robustness and generalization.Figure 6 overviews the constructed datasets.Source: B Datasets · p. 15
B.1 Standard Datasets
FreeMatching’s data strategy combines established correspondence datasets with tracked real-world video, rendered synthetic scenes, and IEG pairs. The sources provide varied motion and geometry supervision alongside data for evaluating identity-preserving correspondence.
B.1 Standard Datasets
- The supervised mixture combines geometric correspondence, optical-flow, and synthetic-video datasets with tracked-video and Blender data.Approximate sampling proportions are rounded to five-percentage-point increments and describe sampling frequency, not dataset size.Source: §B.1 · pp. 15–16, Figure 6 · p. 16
B.2 Real-World Video Dataset
- The real-world video dataset contains 520k ViPE clips annotated with dense pseudo-ground truth from AllTracker, providing varied motion and natural occlusions.Because of data volume and copyright restrictions, the videos and precomputed trajectories are not distributed; preparation instructions and conversion code are provided.Source: §B.2 · p. 16
B.3 Synthetic Blender Dataset
- A synthetic dataset of 360k image pairs uses Objaverse assets rendered from varied viewpoints to provide precise pixel-level correspondence and covisibility supervision.Scenes contain two or three objects with independent rigid transformations, supplying a clean signal for learning occlusion handling.Source: §B.3 · pp. 16–17
B.4 IEG Dataset
- IEG data preparation extracts shared object labels, segments their instances in both images, and unions the masks; training examples can combine two reference–generated pairs.The composite format targets correspondence across multiple objects and cluttered semantics.Source: §B.4 · p. 17
C.1 Benchmarks and Baselines
The appendix specifies benchmark coverage and the training and evaluation protocols, including masked pixel, perceptual, and semantic measures for correspondence quality. It also details the two-stage training setup and the scope of the IEG evaluation.
C.1 Benchmarks and Baselines
- IEG-Bench uses about 100 challenging pairs from UNO’s held-out multi-reference subset, while its single-reference subset is reserved for teacher-guided refinement.Classical tests cover ETH3D and ScanNet-1500 for dense matching and Sintel and KITTI-2015 for optical flow; most use EPE and thresholded pixel-error rates on covisible regions.Source: §C.1 · p. 18
C.2 Supervised Pre-training
- Supervised pre-training combines robust correspondence regression and covisibility-mask accuracy across heterogeneous correspondence datasets.The loss uses Huber and mask terms; the correspondence loss is normalized over valid pixels, and the mask target is encoded in the [-1, 1] range.Source: §C.2 · pp. 18–19
C.3 Teacher-Guided Iterative Refinement
- Teacher-guided refinement starts from the supervised checkpoint and uses UNO+OmniGen2 pairs without dense correspondence annotations.An EMA student supplies initial correspondences and covisibility; frozen RoMa estimates a correction, and the weak objective uses valid pseudo-correspondences plus mask consistency.Source: §C.3 · p. 19
C.4 Implementation Details
- The selected refinement checkpoint is the EMA model after 10k Stage-2 updates, initialized from the supervised checkpoint at step 275k.During the selected run, the transformer and semantic projection are updated while the output decoder and foundation feature extractors remain frozen.Source: §C.4 · p. 19
D Objective Evaluation
- The evaluation reports complementary metrics separately, and the IEG-Bench results use a curated subset of about 100 geometrically complex, diverse image pairs.The evaluation protocol includes pixel-level, perceptual, and semantic measures rather than combining them into one score.Source: D Objective Evaluation · p. 20, §D.4 · p. 21
D.1 Pixel-Level Fidelity: MSE
- Masked MSE measures low-level structural alignment over valid covisible pixels, with reported values scaled by 100.Lower MSE indicates tighter spatial alignment of edges and textures.Source: §D.1 · p. 20
D.2 Perceptual Similarity: LPIPS
- LPIPS captures perceptual similarity that pixel-wise MSE can miss when small misalignments or high-frequency noise do not affect human perception.The protocol uses an AlexNet backbone and a spatial LPIPS map restricted to the covisible region; lower scores indicate better reconstruction.Source: §D.2 · p. 21
D.3 Semantic and Identity Alignment: DINOv3 & SigLIP 2
- DINOv3 and SigLIP 2 feature similarities assess semantic alignment within the masked region, complementing pixel and perceptual measures.DINOv3 averages cosine similarity over valid mask patches; SigLIP 2 provides complementary multimodal semantic alignment, and higher similarities indicate closer feature alignment.Source: §D.3 · p. 21
D.5 Complementary Metrics and Paired Statistics
The section evaluates paired IEG-Bench metrics using shared or fixed masks and complementary feature encoders. It also describes classical-task and IEG comparisons for fixed checkpoints and reports image-level wins across multiple metrics.
D.5 Complementary Metrics and Paired Statistics
- The 10k-adaptation comparison uses the same image pairs, a shared frozen reference mask for Ref-MSE, full images for Full-MSE, and feature encoders distinct from FreeMatching’s DINOv3 features.Ref-DINOv2 uses masked patch similarities, while OpenCLIP compares global image embeddings without masking.Source: §D.5 · pp. 21–22
- FreeMatching’s image-level win rates are 68%, 95%, and 91% against UFM on Full-MSE, Ref-DINOv2, and OpenCLIP, and 73%, 78%, and 87% against RoMa.Paired Full-MSE differences are computed on common evaluation pairs, with 95% confidence intervals from 10,000 bootstrap replicates.Source: §D.5 · p. 22
D.6 Weak Adaptation and Refinement-Domain Controls
- Table 9 compares classical-task EPE for three 10k-IEG-adapted methods, while Table 10 contrasts flow-domain and IEG refinement data sources for a fixed checkpoint.The flow-domain control uses previously unseen, unlabeled Spring/FlyingThings pairs and no ground-truth flow during refinement; the captions identify EPE and IEG-Bench Ref-MSE as reported measures.Source: §D.6 · p. 22, Table 9 · p. 22, Table 10 · p. 22
E Human Evaluation
In a blind human evaluation on IEG-Bench, FreeMatching receives the highest reported mean opinion score among the compared methods. Human ratings also correlate positively with objective DINOv3 scores.
E Human Evaluation
- The blind protocol randomized display order and anonymized methods, with evaluators rating semantic consistency and texture preservation on a discrete four-point scale.The scale ranges from failure or poor alignment to good alignment or a perfect reconstruction.Source: E Human Evaluation · pp. 22–23
- FreeMatching achieves a human MOS of 2.25, exceeding RoMa (1.39) and SEA-RAFT (0.49) in the reported comparison.Ten independent evaluators rated anonymized, randomly ordered results for semantic consistency and texture preservation on a four-point scale.Source: E Human Evaluation · pp. 22–23, Table 11 · p. 23
- Human ratings and objective DINOv3 scores have a strong positive Pearson correlation (r = 0.738).The paper presents this correlation as evidence that its objective metrics align closely with human perception of correspondence quality.Source: E Human Evaluation · p. 23
F Beyond One-to-One Correspondence
For IEG pairs with object replication or synthesis, a query may have multiple plausible correspondences. The paper outlines a probabilistic extension that models these alternatives, while reported experiments use single-pass regression.
F Beyond One-to-One Correspondence
- The FLUX-based framework is extended to model the conditional distribution p(C|Ia, Ib), representing multiple plausible matches for IEG pairs with object replication or synthesis.The reported experiments still use the efficient single-pass regression model.Source: F Beyond One-to-One Correspondence · p. 23
- An iterative denoising objective trains the model to recover clean correspondence coordinates from noisy correspondence states, conditioned on image features and time.The noisy states are formed from correspondence targets according to a noise schedule.Source: F Beyond One-to-One Correspondence · p. 23
- Conditioning on a noisy correspondence state provides a mechanism for representing different matching hypotheses for the same image pair.Source: F Beyond One-to-One Correspondence · p. 23
- Sampling starts from Gaussian noise and iteratively refines predicted clean correspondences; different initial samples can represent alternative hypotheses.Evaluating their validity and diversity requires a dedicated experimental study.Source: F Beyond One-to-One Correspondence · p. 23
G Semantic Matching
Semantic correspondence often aligns different instances of a category, while this task focuses on the identity and local details of the same instance across IEG image pairs. In a multi-object example, FreeMatching preserves more complete shapes and local details than SD-DINO reconstructions.
G Semantic Matching
- Semantic correspondence commonly aligns different instances of a category, whereas this task emphasizes identity and local details of the same instance across IEG image pairs.Source: G Semantic Matching · p. 23
- FreeMatching preserves more complete object shapes and local details than SD-DINO [47], whose displayed reconstructions omit parts and mix object appearance.The SD-DINO evaluation uses the prompts “book” and “vase”; FreeMatching uses only the image pair.Source: G Semantic Matching · p. 23, Figure 7 · p. 24