PaperlayerView exact revision on arXiv

Source-linked AI summary

GLU-Net: Global-Local Universal Network for Dense Flow and Correspondences

Prune Truong, Martin Danelljan, Radu Timofte

arXiv:1912.05524v3cs.CV

TL;DR

Dense correspondence tasks share challenges including large displacements, pixel accuracy, and appearance changes, yet are commonly addressed with task-specific architectures. GLU-Net combines global and local correlations with adaptive resolution in one universal network. Using the same weights, it achieves state-of-the-art performance across geometric matching, semantic matching, and optical flow.

  • Problem

    Dense correspondence applications face large displacements, pixel-accuracy demands, and appearance changes, but existing architectures are specialized for individual tasks.

  • Method

    GLU-Net uses a CNN-based coarse-to-fine architecture combining global and local correlations, plus adaptive resolution to support varied input sizes.

  • Results

    Using the same model weights, GLU-Net achieves state-of-the-art performance for geometric matching, semantic matching, and optical flow.

  • Takeaways & Limitations

    A single universal network can address geometric correspondences, semantic matching, and optical flow within the reported evaluation scope.

  • Takeaways & Limitations

    Global cost volumes have quadratic O((H_lW_l)^2) computation and memory scaling, restricting them to coarse resolutions and constraining input size.

Abstract

from arXiv · show

Establishing dense correspondences between a pair of images is an important and general problem, covering geometric matching, optical flow and semantic correspondences. While these applications share fundamental challenges, such as large displacements, pixel-accuracy, and appearance changes, they are currently addressed with specialized network architectures, designed for only one particular task. This severely limits the generalization capabilities of such networks to new scenarios, where e.g. robustness to larger displacements or higher accuracy is required. In this work, we propose a universal network architecture that is directly applicable to all the aforementioned dense correspondence problems. We achieve both high accuracy and robustness to large displacements by investigating the combined use of global and local correlation layers. We further propose an adaptive resolution strategy, allowing our network to operate on virtually any input image resolution. The proposed GLU-Net achieves state-of-the-art performance for geometric and semantic matching as well as optical flow, when using the same network and weights. Code and trained models are available at https://github.com/PruneTruong/GLU-Net.

1. Introduction

Dense image correspondences support geometric matching, optical flow, and semantic matching, but these tasks face different displacement and appearance challenges. GLU-Net combines global and local correlations to address these challenges across tasks.

  • Dense pixel-to-pixel correspondences underpin applications including visual localization, 3D reconstruction, structure-from-motion, image manipulation, action recognition, and autonomous driving.
  • Geometric matching handles different views with potentially large displacements and appearance transformations, whereas optical flow estimates accurate pixel-wise motion between consecutive frames.
  • Semantic matching finds semantically meaningful correspondences between different instances of the same scene category or object.
  • Local correlations support small-displacement matching, while global correlations capture long-range matches but become computationally costly and resolution-constrained.The complementary limitations motivate combining both correlation types.
  • GLU-Net combines global and local correlation layers to handle large viewpoint changes and appearance transformations while estimating small displacements accurately.
  • The network outperforms previous methods on HPatches and ETH3D geometric correspondences, sets a new semantic-matching state of the art on TSS, and generalizes competitively to KITTI optical flow without retraining or finetuning.

2. Related work

Dense image correspondence research spans classical feature- and variational methods and newer CNN-based approaches. Existing optical-flow networks are effective for small to medium displacements but struggle with strong transformations and substantial appearance changes.

  • Classical correspondence methods use handcrafted or trained feature detectors and descriptors, or variational formulations.
  • Optical Flow: FlowNet introduced trainable CNN-based optical-flow estimation using a U-Net denoising autoencoder trained on FlyingChairs.
  • Optical Flow: PWC-Net and LiteFlowNet use constrained correlation layers on feature pyramids, warping features at each level by the current flow estimate.
  • Optical Flow: These optical-flow networks excel at small to medium displacements with limited appearance change but perform poorly under strong geometric transformations or major visual differences.
  • Semantic Correspondence: DCCNet combines local-feature correlation maps with a context-aware semantic feature representation for semantic correspondence and attribute transfer.

3. Method

GLU-Net is a universal dense-correspondence architecture that combines global and local correlations with adaptive resolution. Its coarse-to-fine design targets long-range matching, precise refinement, and arbitrary-resolution inputs.

  • Problem: GLU-Net estimates dense displacement fields for image pairs without assuming whether they depict different views, consecutive frames, or semantically related content.The displacement field defines pixel-wise motion and is related to the correspondence map m(x) = x + w(x).
  • Correlation layers: Global correlations search all source-target feature locations, whereas local correlations restrict matching to a neighborhood with radius R.The global volume has dimensions H_l × W_l × H_l × W_l; the local volume has dimensions H_l × W_l × (2R + 1)^2.
  • Limitations: Global cost volumes scale quadratically with feature-map size and impose a fixed input resolution, limiting precision on high-resolution images.Their O((H_lW_l)^2) computation and memory cost makes them suitable only at coarse resolutions.
  • Correlation layers: Global correlations capture long-range correspondences at coarse resolution, while subsequent local correlations refine the flow field for precise displacement estimation.Combining both correlation types is intended to support both large and small-range motions.
  • Adaptive resolution: The adaptive-resolution design uses an L-Net at fixed resolution for global matching and an H-Net at original resolution for local refinement.The L-Net provides an initial flow estimate that is upsampled to the H-Net, while intermediate inference refinements can improve accuracy without additional weights.
  • Architecture details: The final GLU-Net contains four pyramid levels: two L-Net levels and two H-Net levels, with global correlation at the coarsest L-Net level and local correlations elsewhere.The implementation uses VGG-16 features; the L-Net operates at 256 × 256, while the H-Net uses the original image resolution.

4. Experimental Validation

GLU-Net is evaluated on geometric matching, semantic matching, and optical flow using the same network and model weights. It combines global and local correlations with adaptive resolution, achieving strong cross-task performance and task-specific gains from added architectural components.

  • Experimental setup: The same network and model weights are evaluated on geometric matching, semantic matching, and optical flow.Training uses DPED-CityScape-ADE, while evaluation covers HPatches, ETH3D, TSS, and KITTI.
  • Geometric matching: 3.6 times faster inference and nearly four times higher PCK-1px than DGC-Net on HP demonstrate benefits from adaptive resolution at high image resolution.The model is already better on HP-240, and its performance gap broadens as resolution increases.
  • Geometric matching: GLU-Net outperforms DGC-Net on HPatches and ETH3D geometric matching, with the advantage increasing for larger image resolutions and displacements.On ETH3D, PWC-Net and LiteFlowNet perform slightly better at low intervals but rapidly degrade as intervals increase; GLU-Net consistently beats DGC-Net in both AEPE and PCK-5px.
  • Semantic matching: Semantic-GLU-Net sets a new TSS state of the art, improving PCK by 1.0% over the previous best, while GLU-Net alone achieves state-of-the-art average performance.The semantic variant adds a consensus network and multilevel feature concatenation specifically for semantic matching.
  • Optical flow: Without retraining or finetuning, GLU-Net achieves the highest AEPE on KITTI-2012 and KITTI-2015 and remains competitive with optical-flow methods.Its larger KITTI-2015 F1 reflects training data that does not model independently moving objects or occlusions.
  • Ablation study: Ablations show that local correlations fail on large displacements, global correlations reduce accuracy, their combination improves performance, and adaptive resolution adds further gains.Iterative refinement particularly benefits high-resolution images; cyclic consistency, NC-Net, and feature concatenation improve HPatches and TSS but slightly degrade KITTI accuracy.

5. Conclusion

The paper proposes GLU-Net, a universal coarse-to-fine architecture for dense flow estimation from image pairs. By combining global and local correlations and using adaptive resolution, it achieves state-of-the-art performance across geometric matching, semantic matching, and optical flow with the same weights.

  • GLU-Net is a universal coarse-to-fine architecture for estimating dense flow fields from image pairs.
  • Combining global and local correlation layers enables long-range displacement estimation while preserving high accuracy.
  • Adaptive resolution addresses the fixed input-resolution constraint otherwise imposed by global correlation.
  • Using the same model weights, GLU-Net achieves state-of-the-art performance on geometric correspondences, semantic matching, and optical flow.

A. Architecture details

The architecture combines global and local correspondence processing in a coarse-to-fine network, with refinement, decoding, and adaptive-resolution mechanisms supporting dense flow estimation.

  • Global correspondence: Global correlation is post-processed with soft mutual-nearest-neighbor filtering to encourage reciprocal matches without adding training weights.The filtered cost volume multiplies each correlation by source- and target-side score ratios.
  • Global correspondence: L2-normalized features produce a global cost volume that the mapping decoder converts into a coarse 2D correspondence map.The map uses normalized coordinates in [−1; 1], is rescaled to image coordinates, and converted into a displacement field.
  • Local flow decoding: The flow decoder estimates residual flow from local correlation and warped source features, then adds the upsampled flow from the preceding pyramid level.At one level, an additional transposed-convolution feature input conveys information from the previous decoder level.
  • Refinement: The refinement network uses seven dilated convolutional layers to refine pixel-level flow before upsampling propagates it to the next pyramid level.Its purpose is to prevent erroneous flows from being amplified across pyramid levels.
  • Adaptive resolution: Adaptive resolution inserts intermediate refinements when the resolution ratio between H-Net and L-Net exceeds three.Intermediate levels are obtained by halving resolution from lH until the ratio to lL is below two, reusing lH decoder weights.
  • Training: Training uses frozen feature-extractor weights, multiscale flow supervision, and unmasked synthetic ground-truth flows, including occluded regions.The loss weights pyramid levels and regularizes network parameters.

B.2. Dataset

The training data are designed to exploit GLU-Net’s adaptive-resolution strategy by using high-resolution images transformed synthetically across several source datasets.

  • Dataset construction: Training is recommended on high-resolution images to use the full potential of GLU-Net’s adaptive resolution strategy.The dataset follows DGC-Net’s procedure while enforcing high resolution.
  • Iterative refinement: Figure 8 illustrates iterative refinement by repeatedly applying H-Net features and weights at intermediate resolutions between L-Net and H-Net.The schematic shows the refinement process across the resolution gap.
  • Dataset construction: The dataset applies 40,000 affine, thin-plate, and homography transformations to higher-resolution images from DPED, CityScapes, and ADE-20K.DPED contributes very large images but only approximately 5,000 image sets from four cameras.

B.3. Implementation details

Implementation uses fixed preprocessing and correlation settings, with separate training configurations for comparison networks and the final GLU-Net.

  • Training configurations: Global-Net and GLOCAL-Net use batch size 32, whereas LOCAL-Net uses batch size 16 with an initial learning rate of 10^-2.The learning rate is gradually decreased, with multiscale weights α1 = 0.32, α2 = 0.08, and α3 = 0.02.
  • Training configurations: GLU-Net uses batch size 16, initial learning rate 10^-4, Adam optimization, learning-rate decay 0.0004, and four multiscale loss weights.The weights are α1 = 0.32, α2 = 0.08, α3 = 0.02, and α4 = 0.01.

C. Detailed results

The supplementary experiments detail runtime, dataset effects, viewpoint-specific evaluation, and additional results across geometric matching and optical flow.

  • Evaluation scope: The supplementary material covers runtime, training-dataset influence, qualitative and quantitative task results, and ablation experiments.Geometric matching, semantic matching, and optical flow are treated in separate supplementary sections.
  • Geometric matching: Table 7 reports AEPE and PCK separately for each viewpoint ID in HP and HP-240.The table provides viewpoint-specific evaluation details for both datasets.
  • Runtime: Runtime is compared on HP-240 using the same desktop with an NVIDIA GTX 1080 Ti, with images resized to 256 × 256 for several networks.The timing averages 295 image pairs.
  • Runtime: GLU-Net has runtime similar to PWC-Net and is three times faster than DGC-Net.The difference is attributed to DGC-Net refining flow to full 240 × 240 resolution while the other networks output quarter-resolution flow.
  • Dataset analysis: Table 6 compares DGC-Net trained on alternative datasets with GLU-Net across geometric-matching and KITTI optical-flow datasets.The comparison includes HP-240, HP, KITTI-2012, and KITTI-2015.
  • Geometric matching: Additional results include detailed HP and ETH3D evaluations, qualitative examples, and analyses of rotation and scaling.These materials extend the geometric-matching analysis beyond the main reported results.

C.3.1 Results on HPatches dataset

GLU-Net outperforms the compared methods across HPatches viewpoints and resolutions, with especially large gains in high-resolution accuracy and low-threshold correspondence precision.

  • GLU-Net outperforms all other methods for every viewpoint ID on both HP-240 and high-resolution HP images.Increasing viewpoint IDs correspond to larger geometric transformations.
  • 3 to 4 times higher PCK-1 on HP is obtained by GLU-Net compared with DGC-Net.
  • GLU-Net achieves better PCK than PWC-Net, LiteFlowNet and DGC-Net at every relative distance threshold.The curves avoid fixed pixel thresholds because HP image pairs have different resolutions.
  • GLU-Net obtains significantly better PCK at low relative distance thresholds.

C.3.2 Results on ETH3D

On ETH3D, GLU-Net maintains correspondence quality as image-pair intervals and appearance changes increase, while specialized optical-flow methods degrade at larger displacements. It also handles very high-resolution scenes and cross-camera differences qualitatively, with robustness extending to substantial transformations but remaining tied to training coverage.

  • Results across image intervals: GLU-Net produces flow fields of constant qualitative quality across increasing ETH3D frame intervals, unlike optical-flow methods that degrade.PWC-Net and LiteFlowNet perform slightly better at low intervals but rapidly worsen as intervals increase.
  • Very high-resolution images: GLU-Net handles very large motions in 24-megapixel ETH3D images, while DGC-Net partly and PWC-Net and LiteFlowNet completely fail.
  • Cross-camera appearance changes: GLU-Net remains robust to illumination and optics changes across simultaneously captured ETH3D camera pairs, producing sharp warped source images.DGC-Net’s warped images are often blurry in these examples.
  • Rotation and scaling: GLU-Net is particularly robust and accurate for rotations up to +/- 50 degrees and scaling factors from 0.8 to 1.4.These transformation ranges correspond to those represented in the training dataset; larger ranges require additional training examples.
  • Optical flow: On KITTI, GLU-Net has better AEPE than PWC-Net and LiteFlowNet overall but is slightly weaker for small displacements.KITTI contains a wider displacement distribution due to moving objects and fast forward camera motion.

C.6. Detailed ablative analysis

Ablations support GLU-Net’s coarse-to-fine design and identify a two-level H-Net configuration as the strongest adaptive-resolution choice for universal performance across matching tasks.

  • Coarse-to-fine approach: AEPE decreases and PCK increases from the coarsest to the highest feature-pyramid level.These results support the use of a pyramidal coarse-to-fine model.
  • Adaptive resolution: Using two H-Net and two L-Net pyramid levels appears best for competitive geometric matching, optical flow and semantic matching.
  • Adaptive resolution: A single L-Net level with three H-Net levels produces poor results on all evaluated datasets.
  • Adaptive resolution: Using only one H-Net level improves HPatches robustness and accuracy but degrades performance on the semantic TSS dataset.
Loading 1912.05524v3…