PaperlayerView exact revision on arXiv

Source-linked AI summary

Segmenting Transparent Objects in the Wild

Enze Xie, Wenjia Wang, Wenhai Wang, Mingyu Ding, Chunhua Shen, Ping Luo

arXiv:2003.13948v3cs.CV

TL;DR

Transparent object segmentation remains difficult because objects inherit diverse background appearance and prior datasets are limited or synthetic. The paper introduces the manually annotated Trans10K real-image benchmark and the boundary-aware TransLab method. TransLab outperforms 20 representative segmentation approaches, while the conclusion states that segmentation in the wild remains far from solved.

  • Problem

    Transparent object segmentation lacks sufficiently large, manually annotated real-world datasets, while transparent objects remain difficult to distinguish from their surroundings.

  • Method

    The paper constructs Trans10K from manually labeled real-world images and proposes TransLab, a boundary-aware segmentation method using boundary prediction to improve segmentation.

  • Results

    TransLab surpasses 20 representative segmentation approaches on Trans10K and outperforms DeepLabV3+ by over 3% mIoU.

  • Takeaways & Limitations

    Trans10K provides a challenging benchmark, while TransLab supports using object boundaries as a cue for transparent-object segmentation.

  • Takeaways & Limitations

    Synthetic data may place transparent objects in semantically unreasonable backgrounds, and real data may lack scale and complexity.

Abstract

from arXiv · show

Transparent objects such as windows and bottles made by glass widely exist in the real world. Segmenting transparent objects is challenging because these objects have diverse appearance inherited from the image background, making them had similar appearance with their surroundings. Besides the technical difficulty of this task, only a few previous datasets were specially designed and collected to explore this task and most of the existing datasets have major drawbacks. They either possess limited sample size such as merely a thousand of images without manual annotations, or they generate all images by using computer graphics method (i.e. not real image). To address this important problem, this work proposes a large-scale dataset for transparent object segmentation, named Trans10K, consisting of 10,428 images of real scenarios with carefully manual annotations, which are 10 times larger than the existing datasets. The transparent objects in Trans10K are extremely challenging due to high diversity in scale, viewpoint and occlusion as shown in Fig. 1. To evaluate the effectiveness of Trans10K, we propose a novel boundary-aware segmentation method, termed TransLab, which exploits boundary as the clue to improve segmentation of transparent objects. Extensive experiments and ablation studies demonstrate the effectiveness of Trans10K and validate the practicality of learning object boundary in TransLab. For example, TransLab significantly outperforms 20 recent object segmentation methods based on deep learning, showing that this task is largely unsolved. We believe that both Trans10K and TransLab have important contributions to both the academia and industry, facilitating future researches and applications.

1 Introduction

Transparent object segmentation is difficult because objects inherit diverse background appearance, while existing datasets are limited or synthetic. The paper introduces Trans10K and TransLab to benchmark and improve this task.

  • Motivation: Transparent objects inherit diverse appearance from their surroundings, making segmentation challenging despite applications in robot manipulation and navigation.Robots must recognize fragile transparent things and fixed transparent obstacles such as glass walls and windows.
  • Trans10K Dataset: Trans10K contains 10,428 manually labeled real-world images collected from complex scenarios with varied scale, viewpoint, contrast, occlusion, category, and transparency.The dataset is reported as 10 times larger than existing datasets.
  • Trans10K Dataset: Trans10K covers diverse scenes, separates transparent things from stuff, and provides 5,000 training, 1,000 validation, and 4,428 test images.Validation and test sets each include easy and hard subsets intended to expose flaws in segmentation algorithms.
  • TransLab: TransLab is a boundary-aware segmentation method that combines a regular content stream, a boundary stream, and Boundary Attention Module feature refinement.The boundary map attends to both high-level and low-level features.
  • Results: 72.1 mIoU versus 69.0 for DeepLabV3+ shows TransLab outperforming a representative state-of-the-art semantic segmentation method on Trans10K’s hard set.The comparison evaluates methods trained on Trans10K.
  • Results: TransLab surpasses 20 representative segmentation approaches, while the dataset and trained models are released to support further research.The paper also reports extensive ablation studies and benchmarking results.

2 Related Work

Prior transparent-object segmentation work uses limited or synthetic datasets and specialized methods, leaving important real-world diversity insufficiently represented. Trans10K is presented as a more diverse and challenging alternative.

  • Semantic Segmentation: Semantic segmentation research commonly uses CNN-based end-to-end fully convolutional networks, with some methods adding structured prediction around object boundaries.Conditional random fields are cited as one structured prediction approach.
  • Transparent Object Segmentation: TransCut segments transparent objects using light-field linearity and occlusion detection, whereas TOM-Net formulates transparent-object matting as refractive-flow estimation.TOM-Net uses a multi-scale encoder-decoder followed by residual refinement and requires refractive-flow labels during training.
  • Transparent Object Datasets: TOM-Net combines 876 real images with 178K synthetic images, repeatedly uses only 4 and 14 objects, and lacks mask annotations for its test set.These properties limit quantitative evaluation and object diversity.
  • Transparent Object Datasets: TransCut contains only 49 images and repeatedly uses 7 objects, mainly bottles and cups.The passage contrasts its limited scale and object coverage with the motivation for newer datasets.
  • Transparent Object Datasets: Synthetic datasets may pair objects with semantically unreasonable backgrounds, while real data can lack scale and complexity.The cited example is a cup appearing to fly with a plane.

3 Trans10K Dataset and Annotation

Trans10K is a large, manually annotated real-world dataset for transparent object segmentation, with fine-grained categories and substantial variation in scene complexity. Its statistics and annotation rules define challenging easy and hard cases for evaluation.

  • 3.1 Data description: The dataset includes transparent things and stuff, with 5,000 training, 1,000 validation, and 4,428 test images.Things include cups and bottles, while stuff includes windows, glass walls, and glass doors.
  • 3.2 Annotation: Annotations use background label 0, transparent things label 1, and transparent stuff label 2, while highly transparent objects are annotated and semi-transparent objects are ignored.Occluded regions are handled differently depending on whether the occluding object is in front of or behind the transparent object.
  • 3.3 Dataset Complexity: Easy samples generally have few, regular, salient, centrally displayed objects, whereas hard samples contain multiple categories, irregular shapes, high transparency, and heavy occlusion.The dataset divides validation and test images into easy and hard subsets according to these properties.

4 Proposed Method

TransLab uses parallel regular and boundary streams, then applies boundary attention to multi-level features to refine transparent-object segmentation. Its training combines segmentation and boundary losses, with Dice loss used for segmentation.

  • Architecture: TransLab uses parallel regular and boundary streams for transparent-object segmentation and boundary prediction.A dilated ResNet50 backbone supports the architecture.
  • Boundary Attention Module: The Boundary Attention Module takes regular-stream features and a predicted boundary map, then refines features through boundary attention and channel attention.The resulting maps are concatenated before channel attention.
  • Decoder: BAM applies boundary attention to C1, C2, and C4 feature maps before progressively fusing high- and low-level features in the decoder.C4 and C2 are fused first, then the result is fused with C1 after up-sampling.
  • Loss Function: The training objective sums segmentation and boundary losses, with λ balancing their relative importance.The experiments set λ to 5; segmentation uses standard cross-entropy and Dice loss.
  • Loss Function: The Dice coefficient compares predicted segmentation with ground truth using corresponding pixel values.Si,x,y and Gi,x,y denote pixels in the segmentation result and ground truth, respectively.

5 Experiments

Experiments evaluate TransLab through implementation details, boundary-focused ablations, and comparisons with mainstream segmentation methods on Trans10K. Boundary supervision and multi-level attention improve performance, while TransLab leads all evaluated baselines across the reported metrics and splits.

  • Implementation: TransLab uses a pre-trained ResNet50, SGD with momentum, batch size 8 per GPU, and 8 V100 GPUs for training.Training lasts 16 epochs with polynomial learning-rate decay.
  • Boundary Loss Selection: Boundary loss improves segmentation as an auxiliary objective across Binary Cross-Entropy, Focal, and Dice supervision.The authors attribute this improvement to multi-task learning that helps the backbone focus on object boundaries.
  • Boundary Loss Selection: Dice Loss achieves the best boundary-stream results without manually adjusting loss hyperparameters.Focal Loss outperforms Binary Cross-Entropy by mitigating boundary-mask sample imbalance.
  • Boundary Attention Module: Using boundary attention at all three feature levels raises mIoU from 70.29% to 72.10% and Acc from 80.38% to 83.04%.The corresponding mBer and MAE also improve, with lower values preferred for those metrics.
  • Comparison to State-of-the-Art: The benchmark includes real-time and regular semantic-segmentation methods, including multiple DeepLabV3+ variants and other mainstream architectures.Methods are evaluated on the challenging Trans10K dataset.
  • Comparison to State-of-the-Art: TransLab outperforms all compared methods on all four metrics across easy/hard splits and things/stuff categories.It surpasses DeepLabV3+ by 3.97% on Acc for the hard set.

6 Conclusion

The paper presents Trans10K as a large real-world benchmark and TransLab as a boundary-aware segmentation method. Benchmarking shows that transparent-object segmentation remains far from solved, while TransLab improves segmentation performance.

  • Conclusion: Trans10K is presented as the largest real dataset for transparent-object segmentation.The dataset is used to benchmark 20 semantic-segmentation algorithms.
  • Conclusion: The benchmark identifies attributes that remain especially difficult for current transparent-object segmentation methods.The authors conclude that segmentation in the wild is far from solved.
  • Conclusion: TransLab uses boundary prediction to improve transparent-object segmentation performance.Its comparisons on Trans10K are framed as validation of the dataset and boundary-attention approach.
  • Conclusion: Visual comparisons report sharper boundaries and higher-quality masks for TransLab than for other semantic-segmentation methods.The stated explanation is its boundary attention, especially in the highlighted yellow-dash region.

A.1 Detailed annotation information.

Trans10K organizes transparent objects into 20 categories and records their distribution across 13 scenarios. Desktop scenes and several object categories are the most frequent, while others are least frequent.

  • Detailed Annotation Information: Trans10K contains 20 transparent-object categories divided into things and stuff, with category frequencies shown separately.Cup is most frequent among things, while French Window is most frequent among stuff.
  • Detailed Annotation Information: The dataset contains 13 scenarios, with Desktop occurring most frequently and Vehicle least frequently among the reported scenes.The passage names the scene as “Vechile.”
  • Detailed Annotation Information: Stationery is the least frequent things category, while Table is the least frequent stuff category.These frequencies are reported alongside the most frequent categories for each object grouping.

Failure Samples Analysis

TransLab generalizes to prior datasets and external imagery, producing high-quality masks, but remains vulnerable in several visually ambiguous scenarios.

  • Highly transparent regions can cause TransLab to fail during segmentation.
  • Strong reflections, overlap or occlusion, and adjacency between semi-transparent and transparent objects can produce confusion or incorrect classification.
  • TransLab outputs very high-quality masks on the prior TransCut and TOM-Net datasets.
  • The method also performs robustly on images captured by mobile phones or collected from YouTube, TikTok, and eBay.

Visual results on external data.

The visual evaluations show TransLab’s qualitative segmentation results across prior datasets, external imagery, and challenging comparisons with other methods.

  • TransLab produces better transparent-object segmentation masks than other semantic segmentation methods in the visual comparisons.
  • Figure 8 presents statistics of the Trans10K dataset, while Figure 9 presents failure cases.
  • Visual examples include transparent-object segmentation results on the prior TransCut and TOM-Net datasets.
  • Additional examples use challenging images captured by mobile phones and obtained from YouTube, TikTok, and eBay.
Loading 2003.13948v3…