PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Joint Task Learning for Generic Object Extraction

Xiaolong Wang, Liliang Zhang, Liang Lin, Zhujin Liang, Wangmeng Zuo

arXiv:1502.00743v1cs.CV

TL;DR

Object extraction must localize salient objects and segment them, but prior methods often treat these tasks independently or search exhaustively. This paper jointly learns two CNNs with latent adjustments between them; it reports superior results on two datasets and faster testing than competing methods.

  • Problem

    Object extraction requires both localization and segmentation, but prior approaches often handle these subtasks independently or rely on exhaustive sliding-window searches.

    Source: §1 · p. 1

  • Method

    Two CNNs jointly perform localization and segmentation, with latent variables adjusting object boxes into segmentation references during EM-type training.

    Source: §1 · p. 2, §4 · p. 4, §4.1 · p. 4

  • Results

    The method reports superior accuracy on the saliency and OE datasets and testing 50–6000 times faster than state-of-the-art methods.

    Source: §5.1 · p. 7

  • Takeaways & Limitations

    The framework handles realistic internet data and can be extended to other joint tasks in similar ways.

    Source: §6 · p. 8

Abstract

from arXiv · show

This paper investigates how to extract objects-of-interest without relying on hand-craft features and sliding windows approaches, that aims to jointly solve two sub-tasks: (i) rapidly localizing salient objects from images, and (ii) accurately segmenting the objects based on the localizations. We present a general joint task learning framework, in which each task (either object localization or object segmentation) is tackled via a multi-layer convolutional neural network, and the two networks work collaboratively to boost performance. In particular, we propose to incorporate latent variables bridging the two networks in a joint optimization manner. The first network directly predicts the positions and scales of salient objects from raw images, and the latent variables adjust the object localizations to feed the second network that produces pixelwise object masks. An EM-type method is presented for the optimization, iterating with two steps: (i) by using the two networks, it estimates the latent variables by employing an MCMC-based sampling method; (ii) it optimizes the parameters of the two networks unitedly via back propagation, with the fixed latent variables. Extensive experiments suggest that our framework significantly outperforms other state-of-the-art approaches in both accuracy and efficiency (e.g. 1000 times faster than competing approaches).

1 Introduction

Object extraction combines localization and segmentation, tasks often handled independently or through exhaustive searches. This paper jointly learns both with two CNNs and latent adjustments to object localizations.

1 Introduction

  • The approach targets generic, category-independent object extraction using a deep joint-task framework.Source: §1 · p. 1
  • The framework uses a localization CNN to predict salient-object positions and scales from raw images, then a segmentation CNN to produce pixelwise masks.This addresses the two sequential subtasks of object extraction.Source: §1 · pp. 1–2
  • Latent variables adjust predicted boxes into segmentation references, addressing cases where localizations omit foreground or include background.The adjustments can enlarge or shrink the boxes passed to segmentation.Source: §1 · p. 2, Figure 1 · p. 2
  • An EM-type procedure alternates between MCMC-based latent-variable estimation and joint network tuning with the references fixed.The segmentation network uses pixelwise errors, while localization tuning accounts for box adjustments.Source: §1 · p. 2

2 Related Work

Prior object-extraction approaches use engineered features, graph-based segmentation, appearance models, or deep networks, while many still rely on sliding-window search. Some joint or co-segmentation methods also require category priors.

2 Related Work

  • Earlier methods combine saliency features with GrabCut, appearance models with spatial or geometric priors, or mask transfer through visually similar sliding-window matches.The cited examples include Cheng et al. and Kuettel et al..Source: §2 · pp. 2–3
  • Object-discovery and co-segmentation methods process image batches but often require category information as priors.Source: §2 · p. 3
  • Several deep detection and segmentation methods still slide windows across image positions and scales, while Erhan et al. predict potential localizations without exhaustive enumeration.Source: §2 · p. 3

3 Deep Architecture

The architecture stacks localization and segmentation CNNs: one predicts a bounding box, and the other extracts a binary mask from the localized image region.

3 Deep Architecture

  • The eight-layer localization network ends with four outputs encoding normalized bounding-box coordinates (x1, y1, x2, y2).Its input image is 224×224×3.Source: §3 · p. 3
  • The segmentation network comprises four convolutional layers and one fully connected layer, producing a 50×50 binary mask from a 55×55×3 input.Source: §3 · p. 3
  • Given an image, the localization network outputs a four-coordinate bounding box, which is used to crop and resize the object region for segmentation.The localization input is resized to 224×224×3; the cropped segmentation input is 55×55×3.Source: §3 · pp. 3–4, Figure 2 · p. 3

4 Learning Algorithm

The learning algorithm jointly trains localization and segmentation networks using latent adjustments to object boxes as segmentation references. It alternates MCMC-based reference estimation with backpropagation updates to both networks.

4 Learning Algorithm

  • Latent variables adjust ground-truth boxes into segmentation references, and an EM-type algorithm estimates them while learning both networks’ parameters.The paper notes that boxes tightly enclosing ground-truth masks may not provide the best references for segmentation.Source: §4 · p. 4, §4.1 · p. 4

4.1 Objective Formulation

  • The objective factors into localization and segmentation probabilities bridged by the adjusted bounding boxes, with separate costs for the two networks.The localization network is trained toward adjusted boxes, while the segmentation network estimates pixel labels from the image and reference.Source: §4.1 · pp. 4–5

4.2 Joint Task Optimization

  • MCMC sampling estimates latent box adjustments using both networks’ costs, with proposals combining a Gaussian term and superpixel-boundary evidence.The proposal uses statistics from previously selected adjustments; its data-driven term favors avoiding box edges that cut through possible foreground superpixels.Source: §4.2 · pp. 5–6
  • After sampling, backpropagation updates segmentation parameters with logistic regression and localization parameters by minimizing squared error to the segmentation references.The algorithm alternates these parameter updates with latent-variable sampling.Source: §4.2 · pp. 5–6, Figure 3 · p. 5

5 Experiment

The experiments evaluate object extraction on the Saliency and OE datasets, comparing accuracy and testing speed with prior methods. They also examine joint learning and cross-dataset generalization.

5 Experiment

  • The Saliency dataset contains 16,233 images, split into 14,233 training and 2,000 test images; the OE dataset contains 10,183 images, split into 8,230 training and 1,953 test images.OE includes objects with more varied appearance and shape or pose, often in cluttered scenes.Source: §5 · p. 6

5.1 Results and Comparisons

  • On the Saliency dataset, Ours(full) reaches 97.81% in P and 87.02% in J, improving on figure-ground segmentation by 5.89% in P and 16.17% in J.Source: §5.1 · p. 7, Table 1 · p. 7
  • On the OE dataset, the method achieves 93.12% in P and 77.69% in J, the highest performances among the compared state-of-the-art methods.Source: §5.1 · p. 7, Table 2 · p. 7
  • Object extraction takes 0.014 seconds per image, versus 94.3 seconds for figure-ground segmentation, 59.6 seconds for CPMC, and 37.4 seconds for Object Proposal.The paper reports that its method is 50–6000 times faster than state-of-the-art methods.Source: §5.1 · p. 7, Table 3 · p. 7
  • Exhaustively enumerating all 625 possible latent-variable proposals for each image achieves similar accuracy to the approach, but costs about 30 times the runtime per training iteration.Source: §5.1 · p. 7

5.2 Empirical Analysis

  • Joint learning improves P by 1.19% and J by 5.92% on the Saliency dataset, and by 1.87% and 6.19%, respectively, on the OE dataset, compared with learning the networks separately.The joint method also achieves lower training costs than the version without latent-variable adjustment.Source: §5.2 · pp. 7–8, Figure 4 · p. 8
  • When trained without Internet-dataset images, the model performs better than two object-discovery and co-segmentation methods on Car and Airplane, and comparably on Horse.The paper reports that this cross-dataset test takes 0.014 seconds per image for its method, versus a few seconds for the co-segmentation methods.Source: §5.2 · p. 8, Table 4 · p. 8

6 Conclusion

The authors conclude that their collaborative deep-network framework handles realistic internet data and extracts object masks efficiently. They also state that it can extend to other joint tasks.

  • The approach extracts object segmentation masks 1000 times faster than competing state-of-the-art methods.Source: §6 · p. 8
  • The joint deep model is reported to handle realistic data from the internet.Source: §6 · p. 8
  • The framework can be extended to other joint tasks in similar ways.Source: §6 · p. 8
Loading 1502.00743v1…