PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTMatte: Boosting Image Matting with Pretrained Plain Vision Transformers

Jingfeng Yao, Xinggang Wang, Shusheng Yang, Baoyuan Wang

arXiv:2305.15272v2cs.CV

TL;DR

Plain vision Transformers had not yet conquered image matting, where high-resolution computation and fine-detail capture pose challenges. ViTMatte adapts ViTs with a hybrid mechanism, convolution neck, and lightweight detail capture module, achieving state-of-the-art results on two benchmarks with fewer parameters.

  • Problem

    Plain ViTs had not yet been established for matting, which poses challenges in high-resolution computation and capturing fine details.

    Source: §1 · pp. 2–3

  • Method

    ViTMatte adapts plain ViTs for matting with a hybrid mechanism, convolution neck, and lightweight detail capture module.

    Source: §5.3 · p. 9, §7 · p. 12

  • Results

    2.54 SAD and 3.06 Connectivity improvement on Composition-1k, and 8.60 SAD and 8.50 Connectivity improvement on Distinctions-646, establish new state-of-the-art results with fewer parameters.

    Source: §1 · p. 3

  • Takeaways & Limitations

    ViTMatte demonstrates the potential of pretrained ViTs for matting with a concise structure and multiple pretraining strategies.

    Source: §1 · p. 3, §7 · p. 12

Abstract

from arXiv · show

Recently, plain vision Transformers (ViTs) have shown impressive performance on various computer vision tasks, thanks to their strong modeling capacity and large-scale pretraining. However, they have not yet conquered the problem of image matting. We hypothesize that image matting could also be boosted by ViTs and present a new efficient and robust ViT-based matting system, named ViTMatte. Our method utilizes (i) a hybrid attention mechanism combined with a convolution neck to help ViTs achieve an excellent performance-computation trade-off in matting tasks. (ii) Additionally, we introduce the detail capture module, which just consists of simple lightweight convolutions to complement the detailed information required by matting. To the best of our knowledge, ViTMatte is the first work to unleash the potential of ViT on image matting with concise adaptation. It inherits many superior properties from ViT to matting, including various pretraining strategies, concise architecture design, and flexible inference strategies. We evaluate ViTMatte on Composition-1k and Distinctions-646, the most commonly used benchmark for image matting, our method achieves state-of-the-art performance and outperforms prior matting works by a large margin.

1. Introduction

ViTMatte adapts plain vision Transformers to image matting, where high-resolution computation and fine-detail capture are key challenges. It reports state-of-the-art results on two benchmarks with fewer parameters than prior methods.

1. Introduction

  • The paper asks whether a task-agnostic pretrained plain ViT can support image matting with concise adaptation, despite high-resolution costs and detail-capture challenges.Matting is framed as precise per-pixel foreground-background separation using an alpha matte.Source: §1 · pp. 1–2
  • Alternating window and global attention, plus a residual convolutional neck, balances computation and matting performance.The authors also use convolution to enhance global attention.Source: §1 · p. 2
  • A detail capture module with fewer than 3M parameters addresses fine-detail information lost through ViT’s fixed patch embedding.The module is a lightweight adaptation for matting.Source: §1 · p. 2
  • ViTMatte saves 70% FLOPs on high-resolution images and achieves better results with fewer parameters than the prior adaptation strategy.The paper also reports new state-of-the-art results on Composition-1k and Distinctions-646.Source: §1 · pp. 2–3
  • 2.54 improvement of SAD and 3.06 improvement of Connectivity on Composition-1k, and 8.60 improvements of SAD and 8.50 improvement of Connectivity on Distinctions-646.The paper reports new state-of-the-art performance with smaller model size.Source: §1 · p. 3

2. Related Work

Prior work motivates plain ViTs for matting through their downstream-task potential, while identifying matting’s detail demands and the open question of leveraging pretrained representations efficiently and accurately.

2.1. Transformer-based Image Matting

  • Transformer-based matting methods have shown encouraging results, but the cited approaches use hierarchical transformers designed as CNN-backbone replacements.The paper studies plain ViTs because matting requires detailed visual information that foundation models may not readily learn.Source: §2.1 · p. 3

2.2. Pre-training of Plain ViTs

  • How to leverage self-supervised pretrained plain ViT representations for matting remains underexplored in both computational cost and accuracy.Examples of such pretraining approaches include MAE, DINO, and iBOT.Source: §2.2 · p. 3

2.3. Plain ViTs for Downstream Tasks

  • Plain ViTs’ non-hierarchical structure limits compatibility with common downstream decoders, yet simple decoder designs have worked in detection and pose estimation.ViTDet uses parallel deconvolutions to form a simple feature pyramid without heavy FPNs.Source: §2.3 · p. 3

3. Methodology

ViTMatte combines a plain ViT backbone with lightweight convolutional adaptations and a detail capture module to predict an alpha matte from an image and trimap. Its training scheme uses trimap-region L1 loss alongside boundary-sensitive losses.

3.1. Preliminary

  • The trimap marks foreground and background so the matting algorithm estimates alpha in the remaining unknown region; the plain ViT is non-hierarchical and produces same-size token features.Source: §3.1 · p. 4

3.2. Overall Architecture

  • ViTMatte concatenates an RGB image and trimap, extracts features with a plain ViT and detail capture module, then fuses features at multiple scales to predict the alpha matte.Source: §3.2 · p. 4

3.3. Vision Transformer Adaptation

  • The backbone uses window attention in most blocks, global attention in the last block of each group, and convolution blocks with residual connections between groups.The hybrid attention reduces complexity for high-resolution images; convolution blocks complement transformer features with high-frequency boundary and texture information.Source: §3.3 · pp. 4–5, Figure 4 · p. 5

3.4. Detail Capture Module

  • The Detail Capture Module uses three simple 3×3 convolution layers to produce detail maps at 1/2, 1/4, and 1/8 resolution, which are fused with upsampled ViT features.Bilinear upsampling and convolution recover the ViT feature to the input resolution; the module supplements finer detail information.Source: §3.4 · p. 5, Figure 5 · p. 5

3.5. Training Scheme

  • ViT weights initialize the original backbone while added components are randomly initialized; the four-channel image-plus-trimap input uses an adapted patch embedding initialized with FNA++.Source: §3.5 · p. 5
  • The training loss combines separate L1 losses on known and unknown trimap regions with Laplacian loss and gradient penalty loss.The region-specific L1 loss is used instead of whole-image L1; the additional losses target boundaries and local smoothness.Source: §3.5 · pp. 5–6
  • Training augmentation applies random affine transformations, crops images to 512 × 512, and jitters hue.Source: §3.5 · p. 6

4. Experiments

ViTMatte achieves state-of-the-art performance on Composition-1k and Distinctions-646, using smaller models than prior methods. The experiments describe the benchmark datasets, evaluation metrics, and training setup.

4.3. Main Results

  • ViTMatte achieves state-of-the-art performance on Composition-1k and Distinctions-646, and ViTMatte-S outperforms prior methods with fewer parameters.On Composition-1k, metrics are measured only in unknown trimap regions; on Distinctions-646, they are measured over the whole image.Source: §4 · p. 6, §4.3 · p. 7

4.1. Datasets and Evaluation Metrics

  • Composition-1k uses 50 foregrounds composited with 20 VOC2012 backgrounds for its 1,000-image test set; its training set has 43,100 composites.Distinctions-646 divides 646 foregrounds into 596 training and 50 test images, producing 59,600 training and 1,000 test composites.Source: §4.1 · p. 6
  • The evaluation uses SAD, MSE, gradient loss, and connectivity loss; lower values indicate higher-quality alpha mattes.MSE values are scaled by 1e-3 for readability.Source: §4.1 · p. 6

4.2. Implementation Details

  • Training composites use trimaps generated by random dilation-erosion kernels of size [30], with RGB and trimap concatenated as model input.ViTMatte-S and ViTMatte-B use ViT-S and ViT-B backbones, initialized with DINO and MAE weights respectively, and trained for 100 epochs on two V100 GPUs.Source: §4.2 · pp. 6–7

5. Ablations and Analysis: Comparisons to Previous ViT-based Tasks

The analysis tests how plain ViTs can be adapted to matting, focusing on attention, convolutional augmentation, and detail features. The experiments support lightweight adaptations that improve performance while reducing computation or model overhead.

5. Ablations and Analysis: Comparisons to Previous ViT-based Tasks

  • The analysis compares ViT adaptation for matting with prior ViT-based tasks and examines structural similarities and differences.The authors describe this as the first direct application of ViT to matting.Source: §5 · p. 7

5.1. Hybrid Attention for Matting

  • Four window-attention layers reduce computation by 50% versus full global attention while improving SAD by 1.52 and MSE by 0.74.The attention maps show stronger activation for window attention than global attention.Source: §5.1 · p. 8, Table 4 · p. 8, Figure 7 · p. 8

5.2. Enhance Transformer Backbones with Convolution Neck

  • The residual bottleneck improves SAD by 1.07 with only 2% extra FLOPs, outperforming the tested convolution blocks.All three tested blocks improve matting performance over the baseline with 2%–5% extra FLOPs; adding convolution blocks after global-attention blocks also improves performance in Figure 8.Source: Table 5 · p. 8, §5.2 · p. 9, Figure 8 · p. 9

5.3. Detail Capture Module

  • Adding DCM to a ViT with one feature map improves MSE by 7.21 with fewer than 0.1M parameters; using D0, D1, and D2 gives the best results.Figure 9 explores DCM depth and reports significant improvement with minimal added parameters.Source: §5.3 · p. 9, Figure 9 · p. 9
  • ViTMatte uses a lightweight convstream rather than SFP to produce high-resolution feature maps, avoiding SFP deconvolution overhead and detail loss.Against independent deconvolutions, it improves SAD by 1.35 and MSE by 0.40 while using 5.7M fewer parameters; for 2048 × 2048 images it reduces computational burden by 71%.Source: §5.3 · pp. 9–10, Table 6 · p. 10

6. Ablations and Analysis: Comparisons with previous matting systems

The section examines pretraining, decoder design, and inference flexibility in ViTMatte. It reports strong performance with self-supervised pretraining, a small decoder, and an inference option that reduces memory with a slight performance penalty.

6.1. Flexible Pretraining Strategies

  • Self-supervised MAE and DINO pretraining yield ViTMatte’s best results, while training from scratch causes significant performance degradation.The experiments also include supervised pretraining strategies.Source: §6.1 · p. 10, Table 7 · p. 10

6.2. Lightest-weight Decoder

  • ViTMatte achieves the highest accuracy with the smallest decoder-parameter count; decoder parameters are 8% of its model, versus 21%–49% for prior approaches.The comparison assesses accuracy, encoder and total model parameters, and relative decoder parameters; the authors suggest a complex decoder may not be necessary with a foundation model.Source: Figure 10 · p. 10, §6.2 · p. 11
  • Without DCM, ViTMatte’s visual results are over-smoothed and lack detail, while its alpha mattes show no obvious semantic errors such as background mapping or region loss.Both ViTMatte variants equipped with the decoder show more visual detail; IndexNet exhibits the cited semantic errors.Source: §6.2 · p. 11, Figure 11 · p. 11

6.3. Flexible Inference Strategies

  • Grid-sample inference groups tokens from each four-token grid before global attention, reducing GPU memory with a slight performance penalty.The strategy is used only at inference; the training strategy remains unchanged, and the table evaluates inference memory on 2048 × 2048 images.Source: §6.3 · p. 12, Table 8 · p. 12, Figure 12 · p. 12

7. Conclusion

ViTMatte is a concise, efficient image-matting system based on plain vision transformers, using a hybrid mechanism and convolution neck. Its lightweight detail capture decoder complements matting details, and the paper reports superiority over a prior ViT adaptation strategy.

7. Conclusion

  • ViTMatte is a concise and efficient image-matting system based on plain vision transformers.The authors report that it is superior to a previous ViT adaptation strategy for matting.Source: §7 · p. 12
  • A hybrid mechanism and convolution neck adapt the plain ViT for image matting.Source: §7 · p. 12
  • The lightweight detail capture module serves as a decoder to complement information required for matting.The paper describes it as the lightest decoder among prior matting methods.Source: §7 · p. 12
  • The authors cite concise structure and varied pretraining strategies among ViTMatte’s advantages over previous matting systems.They hope it will become a standard tool in matting-related industrial applications.Source: §7 · p. 12

Related papers

Loading 2305.15272v2…