PaperlayerView exact revision on arXiv

Source-linked AI summary

Proposal Flow: Semantic Correspondences from Object Proposals

Bumsub Ham, Minsu Cho, Cordelia Schmid, Jean Ponce

arXiv:1703.07144v1cs.CV

TL;DR

Semantic correspondence remains difficult when images vary substantially in appearance and scene layout. The paper introduces proposal flow, which matches multi-scale object proposals using visual and geometric relations, converts sparse matches to dense flow, and evaluates the approach with new benchmarks. Across the PF and standard datasets, the authors report substantial outperformance over existing semantic-flow methods, while noting that PF ground truth is approximate because it is interpolated.

  • Problem

    Semantic flow must establish correspondences across different instances of an object or scene category despite intra-class appearance variation and large scene-layout changes.

  • Method

    Proposal flow establishes region correspondences from multi-scale object proposals using visual features and geometric relations, then transforms the region-based flow into dense correspondences.

  • Results

    Proposal flow substantially outperforms existing semantic-flow methods on the PF benchmarks and further validates these results on Caltech-101, the PASCAL parts, and Taniai’s datasets.

  • Takeaways & Limitations

    The PF datasets support evaluation of both region-based and pixel-based semantic-flow methods without requiring extremely expensive manual annotation of full ground truth.

  • Takeaways & Limitations

    PF ground truth is only approximate because it is obtained by interpolation.

Abstract

from arXiv · show

Finding image correspondences remains a challenging problem in the presence of intra-class variations and large changes in scene layout. Semantic flow methods are designed to handle images depicting different instances of the same object or scene category. We introduce a novel approach to semantic flow, dubbed proposal flow, that establishes reliable correspondences using object proposals. Unlike prevailing semantic flow approaches that operate on pixels or regularly sampled local regions, proposal flow benefits from the characteristics of modern object proposals, that exhibit high repeatability at multiple scales, and can take advantage of both local and geometric consistency constraints among proposals. We also show that the corresponding sparse proposal flow can effectively be transformed into a conventional dense flow field. We introduce two new challenging datasets that can be used to evaluate both general semantic flow techniques and region-based approaches such as proposal flow. We use these benchmarks to compare different matching algorithms, object proposals, and region features within proposal flow, to the state of the art in semantic flow. This comparison, along with experiments on standard datasets, demonstrates that proposal flow significantly outperforms existing semantic flow methods in various settings.

1 INTRODUCTION

Proposal flow addresses semantic correspondence under appearance and layout variation by matching multi-scale object proposals with visual and geometric consistency, then converting sparse matches into dense flow. The paper also introduces challenging benchmarks and reports substantially better performance than existing semantic flow methods.

  • Semantic flow targets images of different instances within the same object or scene category, where appearance and shape variation complicate local matching and geometric constraints.
  • Proposal flow establishes pairwise region correspondences using object proposals and their geometric relations instead of regular-grid local regions.
  • Multi-scale object proposals provide diverse spatial supports, emphasize prominent objects and parts over clutter, and reduce the correspondence search space.
  • Proposal flow exploits visual features and geometric relations to generate region-based matches, which can be transformed into a conventional dense flow field.
  • The paper introduces benchmark datasets and evaluation metrics for both general semantic flow and region-matching methods.
  • Experiments on new and standard datasets show that proposal flow significantly outperforms existing semantic flow methods in various settings.

2 RELATED WORK

Related work spans pairwise and multi-image correspondence, with prior semantic-flow methods relying on regular sampling and spatial regularization. Proposal flow instead uses object proposals, and its pairwise output can initialize multi-image alignment methods.

  • Pairwise correspondence: Classical correspondence methods estimate dense matches between nearby images of the same scene, whereas semantic-flow methods address larger appearance and layout variation.
  • Pairwise correspondence: Existing semantic-flow approaches generally use appearance similarity on regular grids with spatial regularization between neighboring regions.
  • Multi-image correspondence: Proposal flow uses generic object proposals with diverse spatial supports rather than regular sampling, and its output can initialize multi-image methods.
  • Multi-image correspondence: Multi-image alignment methods refine pairwise matches, often using global consistency such as cycle consistency, but at increased runtime.
  • Object proposals and object-centric representations: Object proposals reduce search space and false alarms in detection and have shown repeatability for salient parts, but had been little studied for semantic-flow computation.

3 PROPOSAL FLOW

Proposal flow models region correspondences from object proposals by combining appearance with geometric consistency, then converts sparse matches into a dense flow field. Its local offset matching strategy uses neighboring proposals to reduce clutter-induced errors and support reliable pixel-level warping.

  • 3.1 A Bayesian model for region matching: Proposal flow represents each proposal with appearance and spatial support, then scores candidate matches using appearance similarity and geometric consistency.The best-scoring candidate φ(r) is assigned to each proposal.
  • 3.2 Matching strategies: NAM uses appearance alone, whereas PHM regularizes matches through a global consensus over proposal offsets.PHM aggregates votes from all possible proposal matches, but background clutter can distract the global voting process.
  • 3.2.3 Local offset matching (LOM): LOM estimates one translation-and-scale offset per proposal from overlapping neighboring regions, using a geometric median robust to clutter outliers.The local offset is optimized for each proposal rather than obtained by averaging probabilities over all feasible offsets.
  • 3.2.3 Local offset matching (LOM): LOM imposes local smoothness on neighboring proposal offsets, suppressing clutter matches while favoring regions containing objects rather than object parts.The method uses local geometric relations instead of a global offset consensus.
  • 3.3 Flow field generation: Sparse proposal correspondences become dense flow by selecting each pixel’s highest-scoring neighboring anchor match and transforming coordinates according to the anchor’s position and size.Joint image filtering interpolates locations without direct correspondences, enabling image warping.

4 DATASETS FOR SEMANTIC FLOW EVALUATION

The paper addresses the lack of dense-ground-truth benchmarks for semantic flow by introducing PF-WILLOW and PF-PASCAL, with region-based metrics built on IoU. These datasets use annotated bounding boxes and keypoints to approximate dense correspondences while covering clutter and intra-class variation.

  • Motivation: Existing semantic-flow benchmarks often rely on sparse or indirect ground truth, evaluating only limited matches at keypoints or mask boundaries.Conventional optical-flow benchmarks lack within-class variation, while dense semantic-flow annotation is labor-intensive and subjective.
  • Datasets: PF-WILLOW and PF-PASCAL provide new benchmarks for both region-based and general semantic-flow techniques.PF-WILLOW contains 900 image pairs, while PF-PASCAL contains 1,351 image pairs across 20 object categories.
  • Ground-truth correspondence generation: The datasets use object bounding boxes and keypoint annotations to construct approximate dense correspondences without an extremely expensive annotation campaign.The paper reports that evaluations on these benchmarks predict performance on other tasks and datasets.
  • Ground-truth correspondence generation: Ground-truth region correspondences are approximated by fitting thin plate spline warps to sparse keypoint correspondences and warping proposal-region vertices.Only regions substantially contained within an object bounding box are assigned ground-truth matches.
  • Evaluation criteria: The proposed metrics evaluate region matching precision and retrieval accuracy using IoU between predicted and ground-truth regions.PCR varies an IoU threshold, whereas mIoU@k averages IoU over the top k matches ranked by matching score.

5 EXPERIMENTS

The experiments evaluate proposal-flow components, proposal counts, dense-flow performance, and comparisons with sliding-window and state-of-the-art methods. Results favor object proposals, HOG features, LOM matching, and foreground-focused correspondence, while performance generally saturates near 1,000 proposals.

  • Matching algorithms: 16, 5, and 38 correct matches are found by NAM, PHM, and LOM, respectively, at IoU threshold 0.5 in one cluttered-image example.The comparison attributes LOM’s stronger result to local regularization alleviating background clutter.
  • Proposal and feature comparison: RP gives the best matching precision and retrieval accuracy among evaluated proposal methods on PF-PASCAL with LOM and HOG.The upper-bound plots also indicate that RP generates more consistent regions than the other proposal methods.
  • Proposal and feature comparison: HOG gives the best matching and retrieval performance among the evaluated features when paired with LOM and RP.The comparison notes that ImageNet-trained CNN features may be less suitable for patch correspondence or retrieval than engineered features such as HOG.
  • Proposal and feature comparison: Combining LOM, RP, and HOG performs best on both PCR and mIoU@k across PF-PASCAL and PF-WILLOW.PF-PASCAL has lower matching precision and retrieval accuracy than PF-WILLOW in the reported comparison.
  • Number of proposals: Matching precision and retrieval accuracy increase with proposal count, except for MCG, and begin to saturate around 1,000 proposals.MCG’s inlier fraction decreases as proposal count grows because it produces increasingly more outlier proposals from background clutter.
  • Dense and foreground correspondence: Proposal flow focuses on foreground regions rather than scene clutter, supporting robust matching against outliers.The reported metrics evaluate foreground matching quality separately from irrelevant scene clutter and distracting details.
  • Ground-truth quality: Leave-one-out validation yields average PCK scores of 0.95 on PF-WILLOW and 0.80 on PF-PASCAL for the interpolated ground truth.The authors describe these values as reasonable for validating the generated ground-truth data.
  • Object proposals versus sliding windows: Proposal flow outperforms SIFT Flow, DSP, and DeepFlow, while object proposals outperform sliding-window regions with the same number of regions.On PF-WILLOW, LOM with sliding windows and selective search obtains PCK values of 0.42 and 0.56, respectively.

6 DISCUSSION

The paper presents proposal flow as a region-based semantic-flow method that can produce dense correspondences and introduces PF datasets for evaluation. Across the reported datasets and benchmarks, proposal flow substantially outperforms existing semantic-flow methods, although PF ground truth is approximate because it is interpolated.

  • Discussion: Proposal flow is a robust region-based semantic-flow method that can be mapped to pixel-wise dense correspondences.The method establishes region correspondences using object proposals and their geometric relations.
  • Discussion: The PF datasets provide a benchmark for both region-based and pixel-based semantic-flow methods without requiring extremely expensive manual full-ground-truth annotation.The datasets are presented as a way to evaluate semantic flow using approximate correspondence ground truth.
  • Discussion: Proposal flow substantially outperforms existing semantic-flow methods across the PF datasets and additional standard datasets.The paper reports supporting experiments on Caltech-101, PASCAL parts, and Taniai’s datasets.
Loading 1703.07144v1…