PaperlayerView exact revision on arXiv

Source-linked AI summary

SFNet: Learning Object-aware Semantic Correspondence

Junghyup Lee, Dohyung Kim, Jean Ponce, Bumsub Ham

arXiv:1904.01810v2cs.CV

TL;DR

Semantic correspondence requires dense matching across different object or scene instances despite appearance variation, clutter, and limited pixel-level training annotations. SFNet trains an object-aware CNN using foreground-mask supervision, differentiable matching, and consistency losses, and it significantly outperforms prior methods on standard benchmarks.

  • Problem

    Semantic correspondence needs dense matches across different object or scene instances, but hand-crafted features struggle with semantic variation and clutter while pixel-level labels are costly.

  • Method

    SFNet trains an end-to-end CNN with binary foreground masks, synthetic deformations, kernel soft argmax, mask and flow consistency, and smoothness losses.

  • Results

    SFNet significantly outperforms the state of the art on standard semantic-correspondence benchmarks, including 0.778 average PCK (α = 0.1) on PF-PASCAL.

  • Takeaways & Limitations

    Foreground-mask supervision focuses correspondences on prominent objects while retaining dense, non-parametric flow and requiring no masks at test time.

  • Takeaways & Limitations

    Performance decreases slightly from 0.787 to 0.771 average PCK (α = 0.1) when training expands to MS COCO, mainly because of domain differences from Pascal VOC.

Abstract

from arXiv · show

We address the problem of semantic correspondence, that is, establishing a dense flow field between images depicting different instances of the same object or scene category. We propose to use images annotated with binary foreground masks and subjected to synthetic geometric deformations to train a convolutional neural network (CNN) for this task. Using these masks as part of the supervisory signal offers a good compromise between semantic flow methods, where the amount of training data is limited by the cost of manually selecting point correspondences, and semantic alignment ones, where the regression of a single global geometric transformation between images may be sensitive to image-specific details such as background clutter. We propose a new CNN architecture, dubbed SFNet, which implements this idea. It leverages a new and differentiable version of the argmax function for end-to-end training, with a loss that combines mask and flow consistency with smoothness terms. Experimental results demonstrate the effectiveness of our approach, which significantly outperforms the state of the art on standard benchmarks.

1. Introduction

Semantic correspondence seeks dense matches across different instances of an object or scene, but appearance variation, clutter, and limited annotations make it difficult. SFNet addresses this by training object-aware semantic flow from foreground masks and synthetic deformations.

  • Problem: Semantic correspondence estimates dense flow between images of different instances, where appearance changes, scene-layout variation, and background clutter create major challenges.Classical methods combine feature fidelity with smoothness regularization, but the task remains difficult across substantially different images.
  • Existing approaches: CNN methods provide semantic features and improve correspondence, but semantic-flow approaches require large pixel-level correspondence datasets whose manual annotation is labor-intensive and subjective.Semantic alignment methods instead regress global transformations, which can be sensitive to image-specific details.
  • Proposed approach: SFNet uses binary foreground masks as supervision to learn object-aware semantic flow with a differentiable kernel soft argmax, mask and flow consistency, and smoothness terms.The approach trains end-to-end and does not require masks at test time.
  • Contribution: Foreground-mask supervision offers constraints between semantic-flow and alignment approaches while focusing learning on prominent objects and scene elements.The method produces a dense non-parametric flow field rather than a single global transformation.
  • Results: SFNet significantly outperforms the state of the art on standard benchmarks, with ablation studies analyzing its components and losses.The authors also provide code and models to encourage comparison and future work.

2. Related work

Prior work spans classical and CNN-based semantic correspondence, global geometric alignment, and object-region supervision. SFNet differs by incorporating object-location priors directly into its loss functions while retaining differentiable dense flow.

  • Semantic correspondence: Semantic correspondence methods estimate dense matches between different instances, unlike sparse matching, stereo, or optical flow between nearby views.The broader correspondence literature includes stereo, motion analysis, object recognition, and shape matching.
  • CNN-based methods: CNN features improve semantic representation and robustness to appearance and shape variation, while limited pixel-level annotations motivate synthetic or self-supervised training strategies.Prior methods use ImageNet features, 3D models, or augmented matching data to address training-data constraints.
  • Differentiable flow: SFNet’s differentiable flow field enables end-to-end training with a spatial regularizer.Its architecture computes feature correlations and uses a differentiable matching operation rather than a discrete argmax alone.
  • Geometric alignment: Parametric alignment methods regress global transformations such as affine, homography, or thin plate spline models, applying the same transformation to all pixels.These methods can use synthetically generated data for self-supervised end-to-end learning.
  • Object-aware supervision: Unlike methods that use boxes or masks mainly for preprocessing, SFNet incorporates the object-location prior directly into its loss functions and reports a significant state-of-the-art margin.This design explicitly uses foreground information during optimization rather than only restricting training samples or candidate regions.

3. Approach

SFNet establishes object-aware semantic correspondences by matching dense features with a differentiable kernel soft argmax, then training with mask, flow, and smoothness constraints. The architecture and losses use foreground masks to reduce background-driven ambiguity while preserving end-to-end differentiability.

  • Network architecture: SFNet extracts shared-parameter source and target features, computes all-pairs local-feature similarities, and assigns differentiable matches with a kernel soft argmax.The model is fully convolutional and trained end-to-end because its components are differentiable.
  • Network architecture: Kernel soft argmax restricts averaging around a discrete-argmax location, reducing sensitivity to multimodal matches from background clutter and repetitive patterns.A Gaussian kernel suppresses distant scores while retaining differentiability; the temperature parameter controls concentration and gradient stability.
  • Network architecture: L2 normalization rescales each correlation map before softmax, penalizing features with multiple highly correlated matches and boosting discriminative matches.The matching scores are dot products of local features and form a four-dimensional correlation map.
  • Loss: Binary foreground masks provide an object prior, and the objective combines mask consistency, flow consistency, and smoothness terms weighted by λ_mask, λ_flow, and λ_smooth.Mask consistency is computed by warping each image's mask with the predicted flow and comparing it with the corresponding mask.
  • Loss: Mask consistency discourages foreground-to-background matches, while flow consistency compares forward and backward fields within foreground regions to favor one-to-one matching.Flow consistency helps spread correspondences across foreground regions and alleviates the many-to-one problem left by binary masks.
  • Loss: The smoothness term regularizes flow fields within foreground regions without accounting for correspondences in background areas.This term complements mask and flow consistency by constraining the spatial variation of foreground flow.

4. Experiments

The experiments evaluate SFNet across benchmark datasets, compare training and architectural choices, and show strong performance against existing methods. Ablations identify complementary losses and the kernel soft argmax as important components.

  • Training setup: Training uses synthetically warped image-mask pairs, while PF-PASCAL training uses object bounding boxes because ground-truth foreground masks are unavailable.The experiments use ResNet-101 features and exclude overlapping images from relevant training splits.
  • Experimental setup: The evaluation covers PF-WILLOW, PF-PASCAL, and Caltech-101 using PCK, LT-ACC, and IoU metrics.PF-WILLOW and PF-PASCAL use keypoint-based PCK, while Caltech-101 uses mask-based LT-ACC and IoU.
  • Benchmark results: SFNet outperforms the state of the art by a significant margin on PF-WILLOW and PF-PASCAL, with a PF-PASCAL PCK of 0.79 versus 0.72 for WS-SA.The comparison includes hand-crafted and CNN-based methods.
  • Benchmark results: SFNet outperforms the state of the art in LT-ACC and IoU on Caltech-101 while filtering background clutter without object proposals or inlier counting.These metrics evaluate transformed-mask labeling accuracy and foreground overlap.
  • Ablation study: Combining mask and flow consistency raises average PCK from 0.675/0.718 to 0.782, while adding smoothness further increases it to 0.787.Mask consistency emphasizes prominent objects, whereas flow consistency spreads correspondences across foreground regions.
  • Ablation study: The kernel soft argmax outperforms alternative argmax operators, and using deeper features with all components achieves the best average PCK.The kernel soft argmax supports end-to-end sub-pixel training and is less susceptible to multi-modal matching distributions.

5. Conclusion

The paper presents SFNet for end-to-end object-aware semantic flow with a kernel soft argmax and foreground-mask supervision. The method focuses correspondences on prominent objects and outperforms the state of the art on standard benchmarks.

  • Conclusion: SFNet learns object-aware semantic flow end-to-end using a kernel soft argmax that produces differentiable sub-pixel matches.The architecture uses binary foreground masks to train pixel-to-pixel correspondences.
  • Conclusion: Binary foreground masks provide an accessible supervisory signal for learning pixel-to-pixel correspondences.The paper contrasts these masks with pixel-level annotations, which are harder to obtain.
  • Conclusion: Ablation studies demonstrate the effectiveness of each model component and loss.
  • Conclusion: SFNet is robust to distracting details, focuses on correspondences between prominent objects, and significantly outperforms the state of the art on standard benchmarks.

Supplement

The supplement describes the paper’s organization and points to detailed method, evaluation, alignment examples, and training discussions.

  • Supplement scope: The supplement details the kernel soft argmax and loss functions in Sections 1 and 2.
  • Supplement scope: Quantitative comparisons are reported on PF-PASCAL, PF-WILLOW, and TSS benchmark datasets.
  • Supplement scope: Alignment examples are shown for PF-PASCAL, PF-WILLOW, TSS, and Caltech-101.
  • Supplement scope: The supplement discusses training with bounding boxes and with other datasets.

1. Kernel soft argmax

Kernel soft argmax preserves differentiability while producing matches closer to discrete argmax, especially when feature correlations are multi-modal.

  • 2. Kernel soft argmax: Soft argmax averages spatial coordinates and can produce incorrect matches when multiple features are highly correlated.It approximates discrete argmax reliably only for unimodal matching probabilities with one clear peak.
  • 2. Kernel soft argmax: Kernel soft argmax applies a two-dimensional Gaussian kernel centered at the discrete-argmax position to make matching probabilities approximately unimodal.This design addresses the multi-modal distributions that impair ordinary soft argmax.
  • 2. Kernel soft argmax: Kernel soft argmax approximates discrete argmax well while maintaining differentiability for end-to-end training.The figure contrasts soft and kernel soft argmax correspondences with discrete-argmax matches.

2. Training loss

The training loss combines mask and flow consistency with symmetric flow constraints to prevent many-to-one matches, shrinkage, and incomplete object alignment.

  • 2. Training loss: Mask consistency alone can map multiple source points to one target point because binary masks provide no positional certainty.A single target foreground label may reconstruct the source foreground mask without establishing one-to-one correspondences.
  • 2. Training loss: Flow consistency penalizes inconsistent correspondences and favors one-to-one matching between source and target points.The loss suppresses matches that do not agree with the established flow.
  • 2. Training loss: Computing flow consistency symmetrically for source and target images alleviates flow shrinkage and supports object-to-object matching.Using the loss relative to only one image can cause the flow to contract.
  • 2. Training loss: The loss penalizes inconsistent matches across entire regions, while spreading flow fields helps handle scale changes between objects.Neither individual term handles these cases alone.

3. Quantitative results

SFNet achieves state-of-the-art quantitative performance on PF-WILLOW and PF-PASCAL, with comparisons also reported on TSS.

  • 3. Quantitative results: 19 object categories achieve state-of-the-art results on PF-PASCAL, and the model significantly outperforms all methods on average.The comparison uses average PCK on the PF datasets, with coordinates normalized by image height and width.
  • 3. Quantitative results: Average PCK is the evaluation metric used for quantitative comparison on PF-WILLOW and the PF-PASCAL test split.The reported PCK normalization divides keypoint coordinates by image height and width rather than bounding-box dimensions.
  • 3. Quantitative results: The TSS benchmark contains 400 image pairs across three subsets and seven object categories, with dense flow fields derived from sparse matches and co-segmentation masks.Table 3 reports quantitative comparisons on this dataset.

4. Aligned examples

Qualitative examples across four datasets show dense-flow alignment and transferred labels, including robustness to scale changes and background clutter.

  • 4. Aligned examples: Alignment examples on PF-WILLOW, PF-PASCAL, TSS, and Caltech-101 visualize source images warped to targets using established dense flow fields.The figures present the model's correspondences across multiple benchmark datasets.
  • 4. Aligned examples: The examples show correspondences robust to scale changes between objects and background clutter.Cars illustrate scale changes in PF-WILLOW, while bikes illustrate background clutter in PF-PASCAL.
  • 4. Aligned examples: Caltech-101 examples overlay source and target masks and transfer source pixel labels to target images through established correspondences.This visualization illustrates label transfer using the learned alignment.

5. Discussion

The discussion shows that SFNet remains effective with cheaper bounding-box supervision and limited PF-PASCAL training, but adding substantially more out-of-domain data can reduce performance.

  • Training with bounding boxes: 0.779 average PCK (α = 0.1) with bounding-box masks is comparable to 0.787 using ground-truth masks.The bounding-box model uses the same 2,791 Pascal VOC 2012 images for training.
  • Training with bounding boxes: Flow consistency terms against both source and target images make the method robust to noisy bounding-box labels.They penalize matches between background and foreground regions.
  • Training on PF-PASCAL: 0.778 average PCK (α = 0.1) on PF-PASCAL outperforms reported state-of-the-art scores of 0.72 and 0.68.This result uses object bounding boxes because ground-truth foreground masks are unavailable in the training split.
  • Training on larger datasets: 16,624 MS COCO training images reduce average PCK (α = 0.1) from 0.787 to 0.771, mainly because of domain differences.The larger dataset contains roughly six times as many images as the 2,791-image training set.
Loading 1904.01810v2…