Self-Calibrating Dense Displacement Fields for Reliable Co-Registration of Large Optical Satellite Imagery

Shoukun Sun; Zhe Wang; Sanaz Salati; Jiyin Zhang; Hui Wang; Xiaogang Ma

Source-linked AI summary

Self-Calibrating Dense Displacement Fields for Reliable Co-Registration of Large Optical Satellite Imagery

Shoukun Sun, Zhe Wang, Sanaz Salati, Jiyin Zhang, Hui Wang, Xiaogang Ma

arXiv:2608.22300v1Aug 23, 2026cs.CV

TL;DR

Optical satellite co-registration remains difficult because real image pairs combine heterogeneous differences that violate the fixed motion models, search ranges, and thresholds embedded in existing tools. SCDF instead uses a training-free dense displacement field with a coarse-to-fine predict–measure–filter loop and pair-calibrated filtering. Across 584 constructed-ground-truth problems on real imagery, it produced fields for every cell and stayed within 1.39× of the best result on every real-pair block, while its ablation evidence shows that the full measurement stack matters.

Problem
Existing co-registration methods can fail or silently return grossly wrong fields when heterogeneous image differences fall outside their embedded motion models, while learned matchers lack guarantees outside training distributions.
Method
SCDF estimates a dense per-pixel displacement field through a coarse-to-fine predict–measure–filter loop with pair-calibrated acceptance gates and correlation-based re-measurement and densification.
Results
SCDF produced a field on all 584 cells and was never more than 1.39× from the best result on any real-pair block, while every baseline was somewhere 5.3–783× worse or declined cells.
Takeaways & Limitations
The evaluation indicates that worst-case reliability separates methods more than peak precision when exact ground truth places each method within its deformation model.
Takeaways & Limitations
SCDF was developed and evaluated only on same-optical, same-bandfamily pairs, and cross-modal registration remains untested.

Abstract

from arXiv · show

Co-registration underlies nearly every multi-temporal and multi-sensor use of optical satellite imagery, and operational products still carry documented offsets well above the fraction-of-a-pixel scale at which change detection, time series, and data fusion degrade. Real image pairs differ along several axes at once (sensor response, scene content, viewing geometry, resolution, mosaic seams), and the last of these is not a single global motion. Existing tools embed a motion model and constants tuned to their development data; a pair that fits is registered precisely, while one that does not either fails to match or returns a result wrong by tens of pixels with no failure reported. Learned matchers add a GPU requirement and carry no accuracy guarantee outside their training distribution. We present SCDF (self-calibrating displacement fields), a training-free, GPU-free estimator whose motion model is the dense per-pixel displacement field itself, so no scene motion falls outside the model. A single predict--measure--filter loop runs over a resolution pyramid: the accumulated field predicts where each patch of the moving image falls in the reference, RootSIFT matching and a correlation pass measure the displacement there to sub-pixel precision, and filters whose thresholds are all calibrated on the image pair itself decide what survives. One configuration, with no per-dataset tuning, processes full $8192^2$ scenes on a single CPU core. On 584 constructed-ground-truth pairs built from real Sentinel-2, Landsat-8/9, and NAIP imagery, against seven classical baselines and two zero-shot pretrained matchers, SCDF registers every pair with zero failures, reduces the best baseline's real-pair median end-point error from 6.83 to 4.17m, and cuts its 90th percentile from 17.8 to 7.77m.

I. INTRODUCTION

Optical satellite co-registration must correct residual geometric misalignment across heterogeneous image pairs, yet existing methods can fail catastrophically when their embedded assumptions do not fit. SCDF addresses this with a training-free dense field and pair-calibrated decisions, evaluated on a benchmark designed to expose worst-case reliability.

Motivation: Sub-pixel co-registration is required for change detection, reflectance time series, data fusion, and mosaicking because residual misalignment degrades these analyses.Documented offsets include up to 38 m for Landsat-8 and Sentinel-2 imagery, up to 2.8 pixels for some Sentinel-2 pairs, and 6–19 m in Harmonized Landsat–Sentinel products.
Motivation: Real image pairs differ in radiometry, scene content, viewing geometry, resolution, and mosaicking seams, so no single global motion describes all cases.These differences create conditions that vary across sensor, acquisition, and scene structure.
Limitations of existing methods: Existing tools are precise when their motion models fit but may return nothing or a field wrong by tens of pixels without flagging failure.The strongest baseline completed 554 of 584 problems, yet was at least 10× worse than the best result on 8 of 20 real-pair blocks.
Limitations of existing methods: Learned matchers provide no accuracy guarantee outside their training distribution and require GPU-based tiled inference for 8192^2 scenes.Extending them to new sensor pairings or scale ratios requires representative pretraining or fine-tuning.
SCDF: SCDF uses a training-free dense displacement field, accepts correspondences through pair-calibrated quality and consistency gates, and avoids an imposed global motion model.Its contribution combines the estimator, a 584-cell exact-ground-truth benchmark, and reliability-oriented scoring.

III. METHOD

SCDF represents motion directly as a dense displacement field and estimates it through a coarse-to-fine predict–measure–filter cascade. Re-measurement, densification, interpolation, and warping turn accepted local evidence into a full registration field without global-model commitments.

Output: SCDF outputs a dense displacement field on the moving raster’s pixel grid, with geographic-unit displacements and a companion validity mask.The field is undefined outside the reference–moving overlap and on nodata pixels.
Design principle: The method avoids global motion models and computes every decision threshold from the image pair at the point of use.This removes fixed homography or affine assumptions and absolute acceptance cutoffs from the pipeline.
Cascade: A coarse-to-fine cascade repeatedly predicts patch locations, measures correspondences, filters unreliable points, and carries only the accumulated field to the next scale.Patch size halves at each level, while the field accumulated so far is the sole communication between levels.
Field construction: A correlation pass re-measures matched points, densifies a uniform lattice, interpolates accepted samples into a dense field, and warps the moving image through it.Re-measurement can replace feature matches with corrected displacements, while rejected points may be downweighted rather than deleted.
Design principle: The design predicts stable performance across conditions rather than peak accuracy on any single condition, because it avoids specialized assumptions and regime-specific thresholds.The benchmark tests both the absence of sharp degradation and the expectation that SCDF is rarely best on every individual condition.

C. The cascade: one loop, repeated per level

SCDF applies one self-calibrating predict–measure–filter loop across a resolution pyramid, using the current field to guide matching and local evidence to reject unreliable measurements. Robust statistics and neighborhood consistency protect later levels from propagating bad matches.

Initialization and scale progression: The cascade starts with the full overlap and halves patch size at each level until a 256 px floor, carrying only the accumulated field forward.A single zero-displacement seed is placed at the overlap center, and coarse levels absorb large initial offsets.
Predict: At each patch center, confidence-weighted local affine interpolation predicts a displacement prior and places the reference search window with margin.The prediction uses accumulated points and accounts for local ground-sample-distance differences.
Measure: RootSIFT correspondences are filtered by a ratio test, local RANSAC affine fitting, and forward/backward cycle consistency before becoming confidence-weighted displacement samples.The most confident candidate per pixel is retained in the estimator state.
Filter: Three sequential filters remove gross magnitude outliers and test whether each measurement agrees with a leave-one-out local prediction.The local affine model uses 12 nearest neighbors weighted by confidence and proximity, allowing smooth gradients while rejecting inconsistent points.
Filter: The magnitude gate uses a median-plus-3σ rule derived from the pair’s median absolute deviation, whose 50% breakdown point limits outlier corruption.The factor 1.4826 converts MAD into a Gaussian-noise standard-deviation estimate.

D. The correlation pass: re-measure and densify

SCDF re-measures matched points and densifies the field using one correlation primitive: a predicted displacement is corrected by the sub-pixel correlation residual, then accepted or rejected by pair-calibrated gates. Accepted measurements replace existing values or add points on a fresh lattice, after which the field is interpolated and used for downstream warping.

Re-measure and densify: Re-measurement uses each matched point’s displacement as the prediction and replaces it when a correlation-based correction passes acceptance gates.The same primitive cuts aligned windows, finds the residual peak, and returns a corrected displacement plus a trust verdict.
Re-measure and densify: Densification applies the measurement to a uniform lattice, adding only candidates whose measurements are accepted.Existing matched cells are skipped, and rejected candidates are never added.
Acceptance behavior: Rejected re-measurements retain their prior displacement at reduced confidence rather than deleting anchors at displacement discontinuities.This preserves support near structures such as mosaic seams that a global model cannot represent.
One measurement: The correlation peak s⋆ is the sub-pixel residual added to the prior predicted displacement.The correlation surface Γ(s) scores trial shifts simultaneously, but a peak is trusted only after the acceptance gates.
Field completion: The remaining pixels are filled by an interpolator selected through five-fold cross-validation on the pair’s own control points.The candidates include MLS, inverse-distance weighting, and thin-plate splines.
Field completion: The resulting field is attached to MOV as GDAL geolocation arrays, and every method is rendered through the same downstream warp.This keeps resampling and warp quality from confounding the registration comparison.

B. Constructed ground truth on real pairs

The benchmark injects known displacement fields into real satellite pairs so errors can be scored against exact or controlled ground truth while retaining real sensor, temporal, resolution, and scene differences. E1 isolates construction effects, whereas E2 preserves genuine pair differences and therefore better models deployment, with pre-existing misalignment limiting certifiable accuracy.

Ground-truth construction: Known displacement fields are injected into real image pairs, allowing predictions to be scored against the injected field while retaining real imagery differences.The benchmark covers sensor, temporal, viewing, resolution, and mosaic-related differences rather than a single homography-style motion.
E1: structural test: E1 distorts a reference copy, making the entire REF↔MOV displacement exact and isolating the effect of each construction.It provides an uncontaminated structural test without genuine content change.
E2: realism test: E2 distorts the moving half of real pairs, retaining genuine differences across seasonal, temporal, resolution, and cross-sensor legs.Its five legs include cross-season to five-year gaps, 1:3 and 1:16.7 resolution ratios, and spaceborne-to-airborne imagery.
Evaluation boundary: The E2 pre-existing misalignment r0 is excluded from constructed truth and limits the accuracy that the experiment can certify.This floor is identical for every method, preserving fair comparison; E1 and E2 are not pooled.
Construction families: The four constructions range from sensor changes with zero geometric truth to non-global seam deformation that no single global transform can represent.A2 applies a projective warp, while A4 shifts opposing halves by ±49 px across a transition band.
Scoring protocol: Errors are measured in meters and summarized across cells by medians, with each cell’s p90 reported to expose failure tails.A cell without a field scores +∞ rather than being dropped, so widespread declines appear as FAIL.

E. Baselines and fairness

The evaluation compares SCDF with seven training-free classical baselines under identical inputs and scoring, while pretrained matchers are reported separately because they require adaptation and GPU-based deployment. E1 tests structural assumption violations, and the benchmark reports medians and p90 values while retaining failures.

Baselines: Seven training-free baselines and SCDF run on identical inputs under their official or published defaults and the same evaluator.SCDF uses one configuration across all 584 cells.
Pretrained matchers: SuperPoint+LightGlue and LoFTR are evaluated zero-shot on E1 only and are not ranked against training-free methods.Their adaptation requires a GPU, per-image normalization, tiling, and a separate densification path.
Evaluation design: The frozen matrix contains 5840 runs, scored once under the stated evaluation rule before results are compared.The two experiments answer different questions: E1 tests structural assumptions, while E2 tests real-pair performance.
E1: structural test: E1 block medians range from 0.06 to 7.1 m where a construction does not violate a method’s assumptions, corresponding to 0.006–0.31 px.E1 is a structural test rather than a ranking of precision under exact ground truth.
Fair scoring: The evaluator aggregates per-cell median and p90 EPE in meters and preserves unanswered cells as +∞ rather than rewarding selective failure.A method reads FAIL when declines exceed half the cells in a reported aggregate.
E1: structural test: E1 exposes assumption failures, including 874 m for AROSICS on A2 and 446 m for SIFT+RANSAC on A4.Large entries indicate that the method’s model class has no solution for that construction.

B. Every baseline is a specialist, and every specialist has a cliff

The baselines are specialists whose accuracy depends on whether each pair satisfies an embedded motion model or search envelope. SCDF’s advantage is not peak block accuracy but avoiding catastrophic failures across heterogeneous cells.

Baseline cliffs: Every baseline row contains at least one dark block, each exposing a mechanism the benchmark was designed to test.Figure 6 colors blocks by each method’s ratio to the best result on that block.
Baseline cliffs: 0.062 m is SIFT+RANSAC’s E1/A2 error, but the same method reaches 446 m on E1/A4 and 458 m on E2/A4.Its accuracy is high when the construction is a homography, its embedded model.
Baseline cliffs: 823 m is AROSICS’s E2/A2 error, while its per-window translation model performs well only when deformation is translational.It is at least 10× from the best in 8 of 20 E2 blocks.
Baseline cliffs: 5.3× is GeFolki’s worst block ratio, yet its E1/A2 block median of 0.559 m coexists with a 1424 m per-cell p90.This reflects accurate registration near the warp center and divergence at the corners.
Baseline cliffs: 430–615 m versus 2.3–5.2 m separates global-model methods from dense methods on E2’s same-sensor A4 legs.Opposing seam shifts have no single global-transform solution, so tuning cannot close the roughly 100-fold gap.
Baseline cliffs: 94 of 96 A4 cells are declined by autoRIFT on 8192^2 legs when the seam offset is ±49 px, but it completes 25 of 25 cells when ±16 px fits its fixed 25 px window.Inside that envelope it is best on E2/b4-A4 at 8.84 m.
SCDF’s bounded behavior: 10 of 20 E2 blocks are won by SCDF, while it is top-two in 19 of 20 and has a worst-block ratio of 1.39×.The best baseline’s worst block is 5.3×, whereas SCDF produces a field on all 584 cells.
SCDF’s bounded behavior: 89.9% of E2 cells are within 2× of the best for SCDF, versus 58.3% for GeFolki and 48.1% for AROSICS.Only 1.2% of SCDF cells are more than 10× from the best, compared with 10.1% and 41.3%.

D. What the components buy

The measurement stack provides its main benefit through joint operation, especially in imprecise-match regimes, while SCDF’s broader robustness comes from its self-calibrating skeleton. This dense-field approach has typical dense-estimation runtime, though specialist methods are faster when their assumptions are known.

Measurement stack: Ablating any single correlation-pass or fill stage costs at most 3.5% on the typical cell.The factorial ablation covered a stratified 60-cell subset, with successful cells only.
Measurement stack: 17.1% higher typical-cell error and 41.0% higher tail error result when all three measurement-stack stages are removed together.The joint effect exceeds the sum of individual effects by 14.7%, indicating redundancy among the stages.
Regime dependence: Without the stack, A1 radiometric cells are 1.53× worse on median error and 1.83× worse on p90, whereas A2 homography cells change by 1.004×.This concentrates the stack’s benefit in cells with imprecise matches rather than uniformly across regimes.
Skeleton: Every ablation variant registered all 60 cells, so the bounded worst case is attributed to SCDF’s skeleton rather than any removable measurement stage.The cited skeleton includes recall-first policy, leave-one-out filtering, null-calibrated response gating, and confidence weighting.
Runtime: 963 s is SCDF’s median single-core runtime per 67 Mpx scene, comparable to GeFolki at 992 s and below RIFT2 at 2174 s.SIFT+RANSAC takes 29 s and autoRIFT 73 s, while SCDF can reduce wall-clock time by about 5× with eight workers.
Operating scope: Specialists are preferable when the displacement regime is certified, while SCDF targets heterogeneous or uninspected inputs where fixed assumptions can fail without warning.AROSICS is about 6× faster for guaranteed same-sensor translation, and SIFT+RANSAC or autoRIFT suit known projective or small-displacement cases.

B. Limitations of the method

SCDF’s main limitations arise in open-water scenes, runtime, tested scope, and the constructed-ground-truth basis of its accuracy claims. The authors identify fallback handling, multimodal extension, and runtime engineering as future work.

Open-water limitation: Open-water scenes can cause verification collapse because they provide no persistent structure for matching.On the worst site, only 42 of 433 points verified; the authors propose reverting toward the zero-displacement seed as future work.
Cost and scope: Runtime is unoptimized single-core, requiring approximately 16 minutes per 8192^2 scene.Development and evaluation used same-optical, same-bandfamily pairs; cross-modal registration remains untested.
Benchmark limitation: All reported accuracy numbers rely on constructed ground truth applied to real imagery.Native accuracy floors constrain interpretation: b1–b3 use an approximately 3–5 m literature floor, b4 has a documented bound below 6 m CE90, and b5 supports rankings but no absolute accuracy claim.
Evaluation disclosures: Single low-texture cells can vary by 2–7× under sub-half-pixel perturbations, so no single cell is used as evidence.The disclosures also note a non-functional SRIF reimplementation and autoRIFT’s design-envelope failure.
Future work: Future work includes learned matchers behind self-calibrating gates, true multimodal extension, and runtime engineering.The authors also propose manual-checkpoint evaluation on undistorted cross-sensor pairs and a principled fallback for verification collapse.

A. Internal constants

SCDF uses fixed internal constants across all benchmark cells, while acceptance thresholds are calibrated from batch measurements and separate correlation properties. Its local-affine filtering combines leverage correction, robust reweighting, and fallback handling.

Fixed configuration: All internal constants are identical for the 584 benchmark cells and are not exposed as user configuration.Table S1 lists the pipeline constants.
Local-affine filtering: The leave-one-out local-affine filter predicts each point from its 12 nearest matched neighbors and tests the residual against a local criterion.The model uses distance- and confidence-weighted affine fitting.
Local-affine filtering: Leverage studentization scales residuals before thresholding so extrapolative hull-edge or sparse-area points are not rejected by an interior-calibrated threshold.Degenerate collinear neighborhoods fall back to the weighted mean.
Robust refinement: One Tukey-biweight reweighting pass reduces the influence of points whose residuals corrupt neighboring fits or cross displacement discontinuities.The fits are recomputed once after robust reweighting.
Correlation measurement: Correlation is measured on mean-removed, Hann-tapered windows, then the peak is refined to 1/32 px on a locally upsampled evaluation grid.The windows themselves are never upsampled.
Acceptance gates: The gates separately assess response uniqueness and normalized mismatch because similar open-water windows may lock nowhere while cross-sensor windows may align despite radiometric differences.Response and error therefore encode distinct acceptance properties.
Acceptance gates: Three gates reject measurements failing response, batch-quantile error, or prior-deviation bounds, with thresholds computed only after the batch is measured.The error quantile is clipped, and failed-point confidence may be downweighted relative to the accepted median response.

S-II. BASELINE CONFIGURATIONS AND AUDIT DISCLOSURES

The benchmark compares baselines on identical inputs using official or published defaults and enforced single-CPU execution. Implementations cover global, local, feature-based, optical-flow, and reimplemented methods, with one disclosed non-functional port.

Evaluation protocol: All baselines use official or published defaults, identical inputs, and the same evaluator under enforced single-CPU execution.Thread pools and method-specific parallelism are set to one and runtime occupancy is verified.
Feature baselines: SIFT+RANSAC fits a single global homography to 8000 OpenCV features using a Lowe ratio of 0.75 and a 5 px reprojection threshold.The resulting field is evaluated densely on the moving image grid.
Area and grid baselines: AROSICS uses its default global-plus-local tie-point grid with adaptive window and grid sizes.The configuration is the package default.
Optical-flow baseline: autoRIFT uses official minimal settings, including 32–64 px chips, 32 px spacing, oversampling ratio 64, and a 25 px search limit.Sparse chip displacements are interpolated to the full moving-image grid.
Feature baselines: RIFT and RIFT2 are faithful ports retaining their published FSC outlier rejection, while SRIF is reimplemented from its paper without published source code.RIFT uses affine rejection and RIFT2 similarity rejection.
Implementation consistency: GeFolki uses the official unmodified Python implementation at default settings, and all methods emit dense fields on the full moving-image grid.Four ported methods share a preparation step that warps the reference onto the moving image’s exact grid; a shared routine handles scoring.

B. Disclosure: SRIF port is non-functional

The disclosures distinguish a defective SRIF reimplementation from documented baseline behavior and describe how the constructed benchmark generates and validates displacement fields. Construction assumptions and site-selection rules constrain interpretation of the resulting comparisons.

SRIF disclosure: The SRIF port produces invalid geometry even on near-identity input and is excluded from rankings because it does not represent the published algorithm.The paper attributes the defect to an omitted parameter that required guessing.
autoRIFT disclosure: autoRIFT fails most A4 cells because the seam displacement exceeds its fixed 25 px search window.A grid-size cross-check confirms this mechanism across the reported A4 legs.
Construction formalism: The constructed mover is generated from a base raster and affine grid transform, with a pixel-space displacement field defining both the mover and its ground truth.The ground truth is a map-unit field on the mover’s own grid.
Distortion families: The A1 construction applies radiometric and point-spread-function changes while preserving zero displacement.Its transformations include tone remapping, blur, and re-noising on raw sample values.
Distortion families: The A2 construction uses a corner-perturbed homography, whereas A3 changes resolution with an exactly realized area-averaging factor and zero displacement.The exact grid factor avoids a spurious displacement ramp.
Distortion families: A4 combines a mosaic seam with a weak elastic residual, producing opposite shifts across the image and smoothed random deformation.At 8192^2, the opposing seam shift is approximately ±49 px while the effective elastic amplitude is approximately 0.41 px.
Site selection: Sites are selected by area-weighted sampling and WorldCover composition criteria requiring no dominant class above 90% and multiple substantial classes.This selects scenes with heterogeneous land-cover composition.
Data selection: Imagery comes from Sentinel-2, Landsat, and NAIP collections under cloud and pairing constraints, including same-tile, same-relative-orbit Sentinel-2 pairs.These rules protect the benchmark’s native geometric-accuracy floor.

S-V. COMPONENT STUDIES: GATE ROC AND FACTORIAL ABLATION

The response gate reaches the operating frontier without ground-truth tuning, while factorial ablation shows that the measurement stages are redundant individually but important together, especially for imprecise matches.

A. Response-gate ROC: TPR 0.71 / FPR 0.06 places the self-calibrating response gate on the frontier of an absolute response floor swept against dense ground truth.The gate accepts 71% of good windows and 6% of junk windows without tuning on the cell’s ground truth.
B. Factorial ablation of the measurement stack: A 2^3 factorial ablation switches sub-pixel re-measurement, lattice densification, and cross-validated fill on or off across eight variants.All variants run on the same stratified 60-cell subset, with V0 as the shipped all-on configuration.
B. Factorial ablation of the measurement stack: 17.1% median and 41.0% p90 error increases occur when all three measurement stages are removed, whereas removing any single stage costs at most 3.5% on the median.The stages are mutually substitutable, so one-at-a-time ablations understate their joint contribution.
B. Factorial ablation of the measurement stack: 1.53× median and 1.83× p90 error increases occur for radiometric A1 cells without the stack, while homography A2 cells change only 1.004×.The stack helps where radiometric differences make feature localization imprecise, but re-measurement can add noise on already precise homography matches.
B. Factorial ablation of the measurement stack: 0.717 is the clean-rate for every variant except V4 at 0.683 and V7 at 0.667, indicating that the stages primarily affect precision rather than reliability.No single-stage knockout flips any cell under the supplement’s 0.90 sub-pixel-fraction diagnostic.
C. Zero-shot learned matchers: Zero-shot SuperPoint+LightGlue and LoFTR require grayscale 8-bit tiled inputs and are memory-bound below the full 8192^2 scene size.The adaptation layer averages bands, applies a percentile stretch, and tiles the reference and moving rasters.