PaperlayerView exact revision on arXiv

Source-linked AI summary

Proposal Flow

Bumsub Ham, Minsu Cho, Cordelia Schmid, Jean Ponce

arXiv:1511.05065v3cs.CV

TL;DR

Image correspondence under intra-class variation and large scene-layout changes remains difficult because existing matching can be distracted by appearance differences and clutter. The paper introduces proposal flow, which matches multi-scale object proposals using visual and geometric relations, converts the matches to dense flow, and evaluates them with a new benchmark. Proposal flow substantially outperforms existing semantic flow methods across the benchmark and additional datasets, while the benchmark provides approximate dense-correspondence evaluation without extremely expensive annotation.

  • Problem

    Appearance and shape variation confuse local similarity measures, while semantic-flow evaluation lacks dense ground-truth benchmarks for within-class scene variation.

  • Method

    Proposal flow establishes region correspondences from multi-scale object proposals using visual features and local geometric relations, then maps them to dense flow fields.

  • Results

    Proposal flow substantially outperforms existing semantic flow methods on the PF benchmark and is further validated on Caltech and VOC parts datasets.

  • Takeaways & Limitations

    The PF dataset supports approximate dense-correspondence evaluation of both region-based and pixel-based semantic flow without extremely expensive full-ground-truth annotation.

  • Takeaways & Limitations

    The benchmark assumes that true matches exist only within object bounding boxes.

Abstract

from arXiv · show

Finding image correspondences remains a challenging problem in the presence of intra-class variations and large changes in scene layout.~Semantic flow methods are designed to handle images depicting different instances of the same object or scene category. We introduce a novel approach to semantic flow, dubbed proposal flow, that establishes reliable correspondences using object proposals. Unlike prevailing semantic flow approaches that operate on pixels or regularly sampled local regions, proposal flow benefits from the characteristics of modern object proposals, that exhibit high repeatability at multiple scales, and can take advantage of both local and geometric consistency constraints among proposals. We also show that proposal flow can effectively be transformed into a conventional dense flow field. We introduce a new dataset that can be used to evaluate both general semantic flow techniques and region-based approaches such as proposal flow. We use this benchmark to compare different matching algorithms, object proposals, and region features within proposal flow, to the state of the art in semantic flow. This comparison, along with experiments on standard datasets, demonstrates that proposal flow significantly outperforms existing semantic flow methods in various settings.

1. Introduction

Semantic flow addresses correspondence between different instances of the same object or scene category, where appearance, shape, and layout variation challenge local matching. Proposal flow uses multi-scale object proposals and their visual and geometric relations to produce robust region correspondences, dense flow fields, and a benchmark for evaluation.

  • Semantic flow targets images of different instances within the same object or scene category, beyond the moderate viewpoint changes handled by classical correspondence methods.
  • Appearance and shape variation can confuse local similarity measures, while background, texture, occlusion, and clutter distract existing semantic flow methods.
  • Proposal flow establishes region correspondences from multi-scale object proposals using visual features and geometric relations rather than regular grids.
  • Proposal flow generates region-based semantic flow that can be transformed into a conventional dense flow field for image correspondence and warping.
  • The authors introduce a benchmark for comparing semantic flow methods, matching algorithms, object proposals, and region features.
  • Proposal flow significantly outperforms existing semantic flow methods across the benchmark and standard-dataset experiments.

2. Related work

Object proposals, developed primarily for detection, offer a highly repeatable basis for region matching but had not been thoroughly studied for semantic flow. This work positions proposal choice, matching algorithms, and region features as factors in robust semantic correspondence.

  • Object proposals reduce search space and false alarms in detection and have become important components of state-of-the-art detection pipelines.
  • Their high repeatability on salient part regions makes object proposals useful for region matching, although their role in semantic flow remained insufficiently investigated.
  • The paper introduces proposal flow for robust region correspondences between related but nonidentical scenes using object proposals.
  • The paper introduces a semantic-flow benchmark for evaluating both general semantic flow algorithms and region-matching methods.
  • The study evaluates how object proposals, matching algorithms, and features affect matching robustness and accuracy.

3. Proposal flow

Proposal flow matches multi-scale object proposals using appearance and geometric consistency, then converts region correspondences into a dense flow field. Its local offset matching strategy uses neighboring proposals to reduce the influence of clutter and non-repeatable regions.

  • Bayesian model for region matching: Proposal flow models region matches using both proposal appearance features and spatial-support consistency under the observed proposal sets.The model represents each proposal by an appearance feature and rectangular spatial support, then estimates posterior match probabilities.
  • Geometric matching strategies: NAM uses appearance alone, whereas PHM adds a global consensus over three-dimensional proposal offsets through Hough voting.Global voting can be distracted by background clutter.
  • Geometric matching strategies: LOM estimates a translation and scale offset separately for each proposal using neighboring proposals instead of averaging over all feasible offsets.It defines overlapping-support neighborhoods, gathers neighboring offsets, and estimates a local offset with a geometric median.
  • Geometric matching strategies: LOM imposes local smoothness between neighboring regions, suppressing clutter matches and favoring regions containing objects rather than object parts.The local regularization uses overlap relationships among spatial supports.
  • Flow field generation: Proposal correspondences become a dense flow field by assigning each pixel an anchor match and interpolating its transformed coordinate using the anchor’s position and size.The highest-scoring neighboring proposal supplies the anchor match; joint image filtering fills places without correspondences.

4. A new dataset for semantic flow evaluation

The paper introduces a benchmark for semantic flow because existing evaluations lack dense ground-truth correspondences for images with within-class variation. It generates approximate region correspondences from keypoint-based warping and evaluates them with IoU-based metrics.

  • Existing semantic-flow benchmarks provide only sparse or indirect evaluation because dense semantic ground truth is difficult and expensive to obtain.Conventional optical-flow benchmarks lack within-class variation, while manual semantic correspondence annotation is labor intensive and subjective.
  • The benchmark is designed to evaluate both region-based proposal flow and general semantic-flow techniques with approximate dense correspondence ground truth.Its comparative evaluations are also reported as predictors of performance on other tasks and datasets.
  • Ground-truth correspondence generation: The benchmark uses keypoint annotations and thin-plate-spline warping to approximate dense correspondences between related images.Sparse keypoint correspondences are interpolated with TPS, and pixels are warped from the left image to the right image.
  • Ground-truth correspondence generation: True matches are restricted to object-proposal regions substantially overlapping object bounding boxes, with warped proposals converted into tight rectangular ground-truth matches.The selected subset satisfies |b ∩ r| / |r| ≥ 0.75, and the warped region is enclosed by a tight rectangle.
  • Evaluation criteria: Region matching is evaluated using IoU-based region matching precision and match retrieval accuracy metrics.The metrics are built from IoU between a proposal correspondence and its ground truth, including PCR and average IoU of k-best matches.

5. Experiments

Experiments evaluate proposal-flow components, dense-flow conversion, and comparisons with state-of-the-art semantic-flow methods across multiple datasets. Proposal flow performs strongly, with LOM, RP, and HOG recurring as effective choices, while clutter and deformation remain important factors.

  • Proposal flow components: At IoU threshold 0.5, LOM finds 38 correct matches, compared with 16 for NAM and 5 for PHM in the qualitative example.The accompanying analysis attributes PHM’s weaker performance under clutter to global voting, while LOM’s local regularization alleviates that effect.
  • Proposal flow components: RP achieves the best matching precision and retrieval accuracy among evaluated object proposals, while its upper-bound plots indicate more consistent regions.This comparison uses LOM with HOG and evaluates proposal quality through PCR and mIoU@k.
  • Proposal flow components: HOG gives the best matching and retrieval performance among the evaluated feature descriptors when combined with LOM and RP.The compared CNN features are AlexNet outputs trained for ImageNet classification.
  • Dense flow field: Across the PF dataset, all proposal-flow variants significantly outperform SIFT Flow, DSP, and DeepFlow on dense-flow alignment, with LOM using SS or RP performing best.The authors report that proposal flow is robust to translation and scale changes between objects.
  • Caltech-101: On Caltech-101, proposal flow with LOM outperforms other approaches, especially on foreground-sensitive IoU and LOC-ERR metrics.These metrics separate foreground matching quality from irrelevant scene clutter more directly than LT-ACC.
  • PASCAL parts: On PASCAL parts, proposal flow benefits from cluttered images, while its PCK with SS is similar to other methods when clutter occupies only a small image portion.FlowWeb performs better but uses a cyclic constraint across multiple images, making direct comparison with pairwise matching potentially unfair.

6. Discussion

The paper presents proposal flow as a robust region-based semantic flow method, introduces the PF dataset for evaluation, and reports validation across multiple datasets.

  • Proposal flow establishes robust region-based semantic correspondences and maps them onto pixel-wise dense correspondences.
  • The PF dataset benchmarks both region-based and pixel-based semantic flow methods without requiring extremely expensive full-ground-truth annotation.
  • Experiments on the PF dataset show that proposal flow substantially outperforms existing semantic flow methods.
  • Experiments on Caltech and the VOC parts datasets further validate proposal flow's results.
Loading 1511.05065v3…