PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Transformer Framework for Group-based Segmentation: Co-Segmentation, Co-Saliency Detection and Video Salient Object Detection

Yukun Su, Jingliang Deng, Ruizhou Sun, Guosheng Lin, Qingyao Wu

arXiv:2203.04708v2cs.CV

TL;DR

Separate networks for co-segmentation, co-saliency detection, and video salient object detection have limited transferability and do not fully exploit relationships among group-based features. UFO unifies the three tasks with transformer blocks and an intra-MLP module. The paper reports state-of-the-art performance across the tasks and real-time operation at 140 FPS.

  • Problem

    Earlier approaches use separate networks for the three related tasks, transfer poorly across benchmarks, and can miss long-range feature relationships.

    Source: §I · p. 1

  • Method

    UFO uses one network architecture for CoS, CoSD, and VSOD, combining transformer blocks for long-range group relationships with intra-MLP self-masks to address partial activation.

    Source: §I · p. 2, C. Intra-MLP module · p. 4

  • Results

    UFO reports state-of-the-art performance across CoS, CoSD, and VSOD benchmarks and reaches 140 FPS in real time.

    Source: §I · p. 2

  • Takeaways & Limitations

    The paper presents the same architecture as a unified approach validated across 11 benchmarks spanning the three tasks.

    Source: §I · p. 2, §V · p. 9

Abstract

from arXiv · show

Humans tend to mine objects by learning from a group of images or several frames of video since we live in a dynamic world. In the computer vision area, many researches focus on co-segmentation (CoS), co-saliency detection (CoSD) and video salient object detection (VSOD) to discover the co-occurrent objects. However, previous approaches design different networks on these similar tasks separately, and they are difficult to apply to each other, which lowers the upper bound of the transferability of deep learning frameworks. Besides, they fail to take full advantage of the cues among inter- and intra-feature within a group of images. In this paper, we introduce a unified framework to tackle these issues, term as UFO (Unified Framework for Co-Object Segmentation). Specifically, we first introduce a transformer block, which views the image feature as a patch token and then captures their long-range dependencies through the self-attention mechanism. This can help the network to excavate the patch structured similarities among the relevant objects. Furthermore, we propose an intra-MLP learning module to produce self-mask to enhance the network to avoid partial activation. Extensive experiments on four CoS benchmarks (PASCAL, iCoseg, Internet and MSRC), three CoSD benchmarks (Cosal2015, CoSOD3k, and CocA) and four VSOD benchmarks (DAVIS16, FBMS, ViSal and SegV2) show that our method outperforms other state-of-the-arts on three different tasks in both accuracy and speed by using the same network architecture , which can reach 140 FPS in real-time.

I. INTRODUCTION

Prior group-based segmentation methods use separate networks for co-segmentation, co-saliency detection, and video salient object detection, with limited transferability and difficulty capturing global feature relationships. UFO combines these tasks in one architecture, using transformer blocks and an intra-MLP module; the paper reports state-of-the-art performance and real-time speed.

I. INTRODUCTION

  • Earlier approaches use separate task-specific networks and can perform poorly outside their targeted benchmarks, limiting transferability across tasks.The introduction also notes that convolutional local receptive fields can impede capture of long-range dependencies among pixels.Source: §I · p. 1
  • UFO unifies co-segmentation, co-saliency detection, and video salient object detection in the same network architecture without additional prior information.Its transformer block captures long-range dependencies among group-based image features, while intra-MLP is designed to avoid partial activation.Source: §I · p. 2
  • UFO reports state-of-the-art performance across CoS, CoSD, and VSOD benchmarks and reaches 140 FPS in real time.The reported evaluations cover four CoS, three CoSD, and four VSOD benchmarks.Source: §I · p. 2

II. RELATED WORK

Earlier co-object methods use different matching, clustering, graph, or attention strategies, with limitations that include pairwise scope, computational cost, and weak transfer across tasks. UFO’s encoder-decoder architecture combines multi-resolution transformer co-attention with co-category guidance and intra-MLP self-masks.

II. RELATED WORK

  • Prior CoS approaches include pairwise matching, recurrent group-wise networks, bipartite matching, and clustering, with limitations such as computational cost, restricted scope, or dependence on extracted features.The cited discussion notes that recurrent networks can be computationally expensive, while clustering results depend on upstream features and may make training unstable.Source: §II · p. 2
  • CoSD methods use handcrafted cues, semantic fusion, gradient cues, self-attention, graph networks, or dense correlations, but may miss global group relationships or incur matching costs.The discussion describes graph methods as requiring costly similarity and adjacency matrices, and a dense-correlation method as requiring target/source selection and a sub-solver.Source: §II · p. 2
  • UFO is presented as a unified response to prior methods that do not transfer well across the three group-based segmentation tasks.The related-work discussion says existing attention-based video models do not transfer well to the other tasks.Source: §II · p. 3

A. Architecture Overview

  • The architecture extracts multi-scale features, applies transformer blocks to the last two layers, and combines updated co-attention maps with co-category guidance and intra-MLP self-masks in a pyramid-like decoder.The co-category response is multiplied by decoder features, while the intra self-masks are added; the input may be relevant images or video frames.Source: A. Architecture Overview · p. 3, Figure 2 · p. 3

B. Transformer Block

The transformer block treats image features as patch tokens and uses self-attention to capture global cues across the input patches. Its enhanced representations complement convolutional features and emphasize co-object regions across images.

B. Transformer Block

  • Visual transformers process flattened 2D patches with multi-head self-attention and an MLP, allowing global cues across input patch tokens to be captured in parallel.The paper contrasts this with serial processing methods and notes that parallelization can support real-time performance.Source: B. Transformer Block · p. 3
  • The block flattens group feature maps into patch tokens, projects their dimensions through the transformer, then reshapes the updated tokens into image-like feature maps.The passage defines B and N as batch and group size and describes learned projections between the original C-dimensional features and D-dimensional embeddings.Source: B. Transformer Block · p. 4, Figure 3 · p. 4
  • Transformer features complement convolutional features by enhancing representations of global co-object regions rather than relying only on decisive local features.Visualizations show co-object patches becoming more activated after the block and illustrate long-range matching between image locations.Source: B. Transformer Block · p. 4

C. Intra-MLP module

The intra-MLP module uses within-image patch similarities to activate self-object areas, while network training combines classification and segmentation losses end to end.

C. Intra-MLP module

  • Within each image, the intra-MLP module measures patch similarity, selects each patch’s top-K related patches, and applies MLP and element-wise MAX operations to produce self-masks.The resulting masks are added to decoder features alongside co-category semantic modulation.Source: C. Intra-MLP module · pp. 4–5, D. Network Training · p. 5

D. Network Training

  • Network training integrates classification, weighted binary cross-entropy, and IoU losses to optimize the framework end to end.The WBCE passage defines the ground-truth mask, predicted probability, and positive-pixel ratio used in the loss.Source: D. Network Training · p. 5

IV. EXPERIMENTS

The experiments cover four co-segmentation, three co-saliency, and four video-saliency benchmarks, using task-specific evaluation metrics and shared VGG16 settings where specified.

Datasets

  • Experiments span four CoS, three CoSD, and four VSOD benchmarks, including PASCAL-VOC, CoSOD3k, DAVIS16, FBMS, ViSal, and SegV2.PASCAL-VOC has 1,037 images across 20 categories; CoSOD3k has 3,316 images across 160 categories.Source: §IV · pp. 5–6

Evaluation Metrics

  • CoS is evaluated with Precision and Jaccard index, while CoSD and VSOD use MAE, Fβ, Em, and Sm.Source: §IV · p. 6

Training Details

  • Training uses five-image groups (N = 5), mini-batches of 8 × N, VGG16, 224 × 224 inputs, and four transformer blocks with four multi-heads; K is 4 by default.CoS and CoSD training uses COCO-SEG; for VSOD, the network is initialized with CoS- and CoSD-pretrained weights and trained on DUT.Source: §IV · p. 6
  • Runtime comparisons are tested on a Titan Xp GPU, while the experiments use an RTX 3090 GPU.Source: §IV · p. 6

A. Ablation Studies

Ablations examine module contributions, transformer settings, feature scales, self-mask operations, and matching time. They identify transformer blocks as the main performance contributor while showing that larger configurations can cost computation without consistent gains.

Comparisons to Baseline

  • Each proposed module improves the baseline, with transformer blocks providing the largest contribution across tested combinations.The study varies co-category information, intra-MLP masks, transformer blocks, and their combinations.Source: A. Ablation Studies · p. 6

Settings in Transformer

  • Increasing transformer blocks and heads can improve performance, but continued increases do not always help and cost computation.The authors conjecture that large settings add parameters and redundant information.Source: A. Ablation Studies · p. 6

Multi-Scale Features

  • Using transformer features from both F3 and F4 performs better than using either stage alone, supporting coarse-to-fine cues.Source: A. Ablation Studies · p. 7

Self-Mask Production

  • The intra-MLP and non-local operations improve over standard convolution for self-mask production, with intra-MLP outperforming non-local.Source: A. Ablation Studies · p. 7

Matching Time

  • The proposed method models long-distance co-object similarities while achieving the best matching speed among compared methods.Source: A. Ablation Studies · p. 7

B. Comparison with State-of-the-arts

UFO is compared with state-of-the-art methods across co-segmentation, co-saliency detection, and video salient object detection. The reported results are competitive across tasks, with noted exceptions and dataset-specific limitations.

Co-Segmentation

  • UFO achieves five best and two second-best results across four CoS benchmarks, and is competitive with stronger-backbone methods.It outperforms methods using the same VGG16 backbone, even when some use larger inputs.Source: B. Comparison with State-of-the-arts · p. 7

Co-Saliency Detection

Video Salient Object Detection

  • UFO achieves the best VSOD results on FBMS, ViSal, and SegV2 with VGG16, small inputs, no optical flow, and 140 FPS.It is unsupervised and outperforms some semi-supervised methods.Source: B. Comparison with State-of-the-arts · p. 8, Table VIII · p. 9
  • On DAVIS, co-occurring-object backgrounds such as an audience may be incorrectly segmented as foreground; adding flow cues is straightforward but poor flow slightly lowers FBMS performance.The authors recommend multi-frame images alone, which already achieve state-of-the-art performance in most tasks.Source: B. Comparison with State-of-the-arts · p. 9

V. CONCLUSION

The paper presents a unified group-based segmentation framework for CoS, CoSD, and VSOD, and reports competitive results on 11 benchmarks.

V. CONCLUSION

  • The transformer block and intra-MLP module capture inter-collaborative and intra-activated information in a unified framework for three group-based segmentation tasks.The authors describe this as an early attempt to unify deep-learning networks across tasks and hope it encourages further simplification in other domains.Source: §V · p. 9

A. Details of Using Optical Flow

Optical flow is an optional cue fused with image features to focus on target regions. Additional analyses examine training settings and choices for the intra-MLP and transformer projection, as well as backbone strength.

A. Details of Using Optical Flow

  • Flow cues are combined with image features before the subsequent network, while the transformer, intra-MLP, and decoder remain unchanged; the authors note this fusion may not be optimal.Direct optical-flow input produces comparable predictions, and the method achieves state-of-the-art performance on FBMS, ViSal, and SegV2 without flow.Source: A. Details of Using Optical Flow · p. 13, Figure 8 · p. 13, Figure 9 · p. 14

B. More Training Details

  • For CoS and CoSD, training uses only COCO-SEG without copy-and-paste or Jigsaw augmentation; VSOD training follows prior works’ random flip, rotation, and multi-scale strategy.The VSOD scales are {0.75, 1, 1.25}.Source: B. More Training Details · p. 13

C. More Analysis

  • K = 4 achieves the best performance in the intra-MLP analysis; the authors conjecture smaller or larger values may learn less useful or redundant semantic information.The analysis is identified in Table IX.Source: C. More Analysis · p. 13, Table IX · p. 13
  • D = 782 performs best among the tested transformer projection dimensions; further enlargement brings no gain and consumes computing resources.The tested dimensions are D = 512, 782, and 1024; Table X presents the analysis.Source: C. More Analysis · p. 13, Table X · p. 13
  • A stronger HRNet backbone improves VSOD performance but increases execution time; the authors recommend VGG16, which they report already reaches state-of-the-art performance at real-time speed.They say VGG16 meets most needs in their tasks.Source: C. More Analysis · pp. 13–14, Table XI · p. 14

D. More Downstream Applications

The paper illustrates additional uses of its framework in co-localization, real-time video saliency, and bullet-chat blocking. These examples are presented through visual demonstrations rather than reported benchmark comparisons.

Co-Localization

  • On CUB-2011, generated co-object masks are converted to predicted bounding boxes that the authors describe as close to ground truth.Figure 11 identifies input images, predicted masks, and localization results by row, with ground-truth boxes in red and predictions in green.Source: D. More Downstream Applications · p. 14, Figure 11 · p. 15

Real-Time Video Salient Object Detection

  • A randomly selected public video demo shows salient-object detection without YouTube-VOS training; the authors describe its result as competitive.The demo result is provided through the project code page.Source: D. More Downstream Applications · p. 14

Bullet-Chat Blocking

  • For bullet-chat blocking, the authors use OpenCV to overlay pseudo bullet-chat text sliding across a public video and produce a blocking demo.The resulting video is provided through the project code page.Source: D. More Downstream Applications · p. 14

E. More Visualizations

The visualizations show feature matching and predicted outputs across the paper’s co-segmentation, co-saliency, and video salient-object tasks. The authors use matching visualizations to illustrate how the network identifies similar object regions.

Matching

  • Highly correlated patches across group-based images illustrate the network’s matching of object similarities and discovery of object regions.Figure 10 visualizes the object-matching flow.Source: E. More Visualizations · p. 15

Co-Segmentation

Co-Saliency Detection

Video Salient Object Detection

Related papers

Loading 2203.04708v2…